Slashdot Mirror

← Back to Stories (view on slashdot.org)

Linode Under DDoS Since Christmas (linode.com)

Posted by Soulskill on from the angry-people-creating-other-angry-people dept.

hol writes: Linode has been getting hit with DDoS attacks since Christmas Day, and it looks like their pain is set to continue. The attackers are rotating DDoS traffic through various regions of Linode's service. They say, "All of these attacks have occurred multiple times. Over the course of the last week, we have seen over 30 attacks of significant duration and impact. As we have found ways to mitigate these attacks, the vectors used inevitably change. As of this afternoon, we have mostly hardened ourselves against the above attack vectors, but we expect more to come. ... Once these attacks stop, we plan to share a complete technical explanation about what has been happening." See their status page for updates.

18 of 149 comments (clear)

  1. Oh no! by Anonymous Coward · · Score: 5, Interesting

    WTF is "Linode"?

    1. Re:Oh no! by ArchieBunker · · Score: 4, Insightful

      The janitors who run this site can't even be bothered to hyperlink Linode to Wikipedia.

      --
      Only the State obtains its revenue by coercion. - Murray Rothbard
    2. Re:Oh no! by Snotnose · · Score: 4, Interesting

      Exactly. I've never heard of Linode and I run Linux on one of my machines. Is it too much to ask that you give a 1-2 sentence description of why I should care?

      And no, I don't care enough to check wikipedia, nor google. Never heard of 'em, don't care, the summary gives me no reason to change my mind on either.

    3. Re:Oh no! by rudy_wayne · · Score: 3, Funny

      Since your Google appears to be broken

      https://www.linode.com/

      They are a "Cloud Hosting" company. Which makes the recent events quite the Lulz.

    4. Re:Oh no! by Razed+By+TV · · Score: 5, Insightful

      This is becoming a regular problem on /. Article titles and summaries are increasingly assuming that people have the obscure knowledge of the topic to actually care. In this case, who Linode is, what makes Linode important, why this DDoS merits more attention than other attacks, etc.

      It used to be that when I saw a title/summary that I was unfamiliar with, I could follow it and expect to learn something from it.
      Now I find out that JustAnotherCompany experienced JustAnotherThingThatHappensOnTheInternet.

      I googled Linode, so I guess I learned something. Cloud hosting/virtual servers. Are they big fish, little fish, do they host someone big, are they known for something they did in particular? Well, I have better things to do than research it.

    5. Re:Oh no! by Anonymous Coward · · Score: 3, Insightful

      This site is run by janitors? That actually explains a lot.

      No it doesn't; janitors clean up the shit that overflows everywhere

    6. Re:Oh no! by h33t+l4x0r · · Score: 5, Informative

      I believe they're the number 2 player after AWS (Amazon Web Services). So a big fish, and it's an impressive accomplishment to give them so much trouble.

    7. Re:Oh no! by vel-ex-tech · · Score: 5, Informative

      Linode is a quite good VPS provider. They have several stock distro installs to choose from (Linux and BSD), and then the sky is the limit. They also pay for user-generated documentation, and the focus is on FLOSS software that you can install and configure on your node. This isn't some PHP MySQL crap. I've been a happy user for years now, running a private mail, web, and IRC server. The prices are quite than reasonable. I'm not sure if they offer Xen nodes anymore since KVM is the way to go.

      My nodes at Fremont haven't been affected yet. Soylentnews, also hosted on Linode, seems to be doing well too.

  2. Haven't noticed a thing... by Anonymous Coward · · Score: 3, Informative

    As one of their customers, I haven't noticed a thing. My instances keep on chugging along as if nothing is wrong.

    1. Re:Haven't noticed a thing... by DRichardHipp · · Score: 5, Informative
      https://www.sqlite.org/ is hosted on Linode - has been for over 10 years. The site was off-line for about 10 minutes on Tuesday, but service has been OK otherwise.. The folks at Linode have done a good job of keeping things running. I see now that Chris Aker and his team have had a challenging week.

      I've used a variety of hosting providers, but I always keep coming back to Linode. Their product is competitively priced, they provide exceptional service and support, and they are very simple to use. And, unlike AWS, you don't need a calculator and 2 hours spent parsing fine print in the documentation to figure out how much a given level of service will end up costing you. I highly recommend Linode for your cloud computing needs. I hope they are able to resolve their DDoS problems quickly.

    2. Re:Haven't noticed a thing... by devilspgd · · Score: 4, Interesting

      I've got several Linodes, I've probably seen about 10-15 minutes of downtime total (per node, and not at the same time), so in my case this translated into approximately 8 minutes of customer-facing outage due to my internal redundancy.

      However, my redundancy is within a Linode network, if an entire Linode data-center goes down, so do I, I don't attempt to replicate outside of an individual DC, outside of off-site backups (which I store outside the Linode environment). We do have core infrastructure (DNS, our own mail and system status pages) distributed across multiple providers so that losing a single provider won't take us down, although this is mainly to prevent a situation such as where my Linode account itself is suspended.

      All in all, I'm quite impressed at how well they've handled it.

      --
      Give a man a fish, he'll eat for a day, but teach a man to phish...
    3. Re:Haven't noticed a thing... by srw · · Score: 3, Informative

      I was wondering why siteuptime was reporting brief outages on my Linode. I guess this explains why I've had the worst uptime in years... just got my December report. 99.66% uptime. I've been with Linode for a very long time and can honestly not say a bad word about them. I've really only ever had one bad outage, when the UPSes at HE Dallas blew up. I think that one ended up being about 8 hours. They kept us updated and even set up a webcam so we could watch them working in the datacentre getting things booted back up after power was restored.

  3. Maybe I'm a jerk... by devilspgd · · Score: 4, Interesting

    Okay, I'm probably a jerk, but I don't care and I hope their upstream(s) isn't/aren't helpful.

    I'm a happy Linode customer, but when one of my customers was being targeted by a DDoS extortion scheme which was using a very specific, very blockable attack, Linode's only solution/suggestion was to boot the customer, or wait it out, and in the meantime, they nullrouted my IP. Now I get that nullrouting my IP keeps the rest of the customers in that subnet/node/etc online, but it frustrated me that they wouldn't even attempt to block selectively, and as such, I can't get a ton of sympathy when they're victims of similar attacks.

    And for the record, my customer didn't pay, eventually the DDoS group got bored and moved on.

    --
    Give a man a fish, he'll eat for a day, but teach a man to phish...
    1. Re:Maybe I'm a jerk... by Anonymous Coward · · Score: 3, Interesting

      Well, hopefully you won't mind me sounding like a jerk too, but I'm going to outright call you out as a liar on that one plain and straight.

      As a Linode customer for over a decade now with a good number of nodes still running, and who has previously been the target of DDoS attacks, I have never seen caker or any of their staff respond in such a way to one of their customers.
      They have been nothing but above and beyond helpful with blocking attacks and working with me to keep my (and their own) services up, including custom layer 3 filtering at their upstream providers specifically to address the types of attacks directed at me.

      Yes they null route the specific IPs being attacked during the peek of the attacks to keep services up for their other customers first, and you expecting anything different is not only foolish but demonstrates unrealistic expectations and a failure to grasp how network routing works at the backbone level.

      But to claim they threatened either you or your customer must go has no president, and does not mesh with how they have always handled such things in the past.

      Did you go about threatening them with lawsuits like an entitled prick or something? Maybe then I could see such a response (and not really blame anyone for it)
      But you imply trying to work with them on the problem so I doubt that to be the case, which is why I doubt your explanation of what happened is at all the full story.

      All of that aside however, how fucking petty and low to wish such DDoS attacks on them for no other reason than you being a little frustrated at being offline for a few minutes expecting god like powers from those who clearly don't have them.
      Especially when their entire staff has worked around the clock over the holidays to keep services of entitled pricks like you running instead of being with their friends and families, all over a petty exaggerated grudge.

    2. Re:Maybe I'm a jerk... by mysidia · · Score: 5, Informative

      Null routing an IP address under DDoS attack in an emergency is standard industry practice across all major ISPs and hosting providers; companies that use more advanced techniques either have a few tricks up their sleeves which only work in the most common situations, or they bought some $5 million anti-DoS appliances to help mitigate it (usually).

      The simple fact is DoS mitigation is not part of a basic hosting service, once an attack exceeds a few million packets per second, or a couple Gigabits: you are simply not paying network providers enough money for it to be feasible for any ISP to come close to justifying effective DoS mitigation for those rare sizes of attack, for every customer, because the cost involves provisioning hundreds of million$$$ in extra upstream capacity, internal network capacity, and operations staff.

      Then even with all that extra capital spend: (1) It's still not possible to make every attack seamless, Null-routing might still be required in cases, there will still be outages, people like the above will still be unhappy, And.... (2) Most ISPs don't have that much throwaway cash, and most hosting customers aren't going to be willing to pay their share of what it costs to provision 10000x as much capacity as needed.

      (3) Its less expensive to just shed overly-demanding customers who pay little by allowing them to make themselves unhappy and go to a competitor. If someone's paying $100 a month and their site is constantly getting DDoS'd, then it makes perfect sense to terminate them as a customer to, and let the other 10000 $100/Month customers have a better experience, instead of leaving due to the DoS being suffered as a result of 1 customer.

      And if someone wants to arrange for their website to be handled differently, then this is part of a negotiation that should be made with the ISP or provider before turning up hosting service and added to the contract, with response SLA and recourse/refund policy.

      Or you're better off enlisting a 3rd party DoS-scrubbing service such as CloudFlare to conceal your infrastructure from attackers.

      There are also DoS-cleanup services that work at a network range level where your DDoS provider announces your /24 into BGP, cleans DoS, and forwards you traffic.

      Many ISPs do have the flexibility for alternate handling of DDoS, up to a certain point, they can avoid Null-routing an IP, or avoid the Null-routing of one IP from making your service unavailable.... generally, the cost will be much higher --- E.g. $10,000 per month instead of $100 per month.

      Forget about attempting to negotiate expert-level DoS management that will require the provisioning of engineer and infrastructure resources in advance that are quite costly to the providers to keep on hand, Unless you are willing to pay sufficiently to be a large client of the provider with a multi-year committed contract and cover the costs of those extra resources plus sizable profit.

      Also: to host a website resiliently, however, the provider will most likely require that the website be served from multiple server farms in multiple IP ranges with an anycasted internet presence for both the services' IP addresses, and the supporting DNS services.

      This is because in spite of additional resources, it might still be necessary at times to fall back to Null-routing.

    3. Re:Maybe I'm a jerk... by devilspgd · · Score: 3, Interesting

      I see why you're posting as A/C. Try reading again: Linode didn't threaten me or the customer or anyone else, we (my customer, and me as a host) were threatened by a DDoS extortion scheme -- A "Nice web site you host there, it would be a shame if someone were to keep up a sustained DDoS against it"

      And no, I didn't threaten a lawsuit or anything else, I just asked them for information on the type of attack, and later once I identified the type of attack, help filtering rather than a complete nullroute.

      They were polite, but completely unhelpful in terms of even providing any information about the particulars of the attack (one source or multiple, port numbers, type of traffic, etc) "Luckily" the DDoS hit servers I run on another network too, and the network operator there was able to provide me with said details, which helped to mitigate the attacks.

      --
      Give a man a fish, he'll eat for a day, but teach a man to phish...
    4. Re:Maybe I'm a jerk... by devilspgd · · Score: 4, Informative

      I didn't wish such DDoS attacks on them at all. Has /. reading comprehension really fallen this low?

      What I hope is that their provider is as unhelpful to them as Linode was to me when I was a victim of similar, ongoing and sustained attacks, as it will help them understand the difficulty that customers face and that they're left struggling to resolve it on their own because if so, they may develop both sympathy and tools that can be used to protect both themselves and their customers in the future.

      If "Oh, just shut everything down and wait it out" is good enough for me, it should be good enough for them. If not, well, maybe they'll improve after having a bit more personal experience being the victim.

      And for the record, I'm still a Linode customer (and have more services with them now than I did then); I was just disappointed at their lack of usefulness.

      --
      Give a man a fish, he'll eat for a day, but teach a man to phish...
  4. Re:Why? by mysidia · · Score: 3, Insightful

    These attacks cost money. People don't do them for fun anymore.

    You are sadly mistaken.... just b/c attacks cost money, does not mean people do not still launch them willy-nilly ---- you are a hosting provider, and some website you are hosting makes the wrong person or wrong group angry, or they have a political message to send, they may attack the entire provider.

    Although, quite often such large-scale attacks are launched, then followed up by "ransom demands", and as long as the target does not pay the huge cash ransom, the attacks continue, And the attacker stands to make much more $$$ from ransom than the attacks would have cost them.