Slashdot Mirror

← Back to Stories (view on slashdot.org)

Linode Under DDoS Since Christmas (linode.com)

Posted by Soulskill on from the angry-people-creating-other-angry-people dept.

hol writes: Linode has been getting hit with DDoS attacks since Christmas Day, and it looks like their pain is set to continue. The attackers are rotating DDoS traffic through various regions of Linode's service. They say, "All of these attacks have occurred multiple times. Over the course of the last week, we have seen over 30 attacks of significant duration and impact. As we have found ways to mitigate these attacks, the vectors used inevitably change. As of this afternoon, we have mostly hardened ourselves against the above attack vectors, but we expect more to come. ... Once these attacks stop, we plan to share a complete technical explanation about what has been happening." See their status page for updates.

32 of 149 comments (clear)

  1. Oh no! by Anonymous Coward · · Score: 5, Interesting

    WTF is "Linode"?

    1. Re:Oh no! by ArchieBunker · · Score: 4, Insightful

      The janitors who run this site can't even be bothered to hyperlink Linode to Wikipedia.

      --
      Only the State obtains its revenue by coercion. - Murray Rothbard
    2. Re:Oh no! by Snotnose · · Score: 4, Interesting

      Exactly. I've never heard of Linode and I run Linux on one of my machines. Is it too much to ask that you give a 1-2 sentence description of why I should care?

      And no, I don't care enough to check wikipedia, nor google. Never heard of 'em, don't care, the summary gives me no reason to change my mind on either.

    3. Re:Oh no! by rudy_wayne · · Score: 3, Funny

      Since your Google appears to be broken

      https://www.linode.com/

      They are a "Cloud Hosting" company. Which makes the recent events quite the Lulz.

    4. Re:Oh no! by h33t+l4x0r · · Score: 2

      The janitors who run this site can't even be bothered to hyperlink Linode to Wikipedia.

      This site is run by janitors? That actually explains a lot.

    5. Re:Oh no! by Razed+By+TV · · Score: 5, Insightful

      This is becoming a regular problem on /. Article titles and summaries are increasingly assuming that people have the obscure knowledge of the topic to actually care. In this case, who Linode is, what makes Linode important, why this DDoS merits more attention than other attacks, etc.

      It used to be that when I saw a title/summary that I was unfamiliar with, I could follow it and expect to learn something from it.
      Now I find out that JustAnotherCompany experienced JustAnotherThingThatHappensOnTheInternet.

      I googled Linode, so I guess I learned something. Cloud hosting/virtual servers. Are they big fish, little fish, do they host someone big, are they known for something they did in particular? Well, I have better things to do than research it.

    6. Re:Oh no! by Anonymous Coward · · Score: 3, Insightful

      This site is run by janitors? That actually explains a lot.

      No it doesn't; janitors clean up the shit that overflows everywhere

    7. Re:Oh no! by Anonymous Coward · · Score: 2, Informative

      They're an inexpensive high performance VPS (virtual private server) provider. I've used Linode myself to roll my own VPN (virtual private network) for example. Many major companies and various nerds use Linode to host VPS running on SSD (solid state disk) storage in a KVM (Kernel-based Virtual Machine), they can handle larger loads of users than many other cloud services. And, they tend to be cheaper because they charge you a flat rate instead of nickel and diming you for every cpu/ram/io usage. They give you 2TB throughput, 1gb ram, 1 cpu, 24 gb ssd, 40gb/125gb in/out for $10/mo and these stats scale up evenly for every extra $10/mo. I've used their service, it is pretty good. DigitalOcean is one of their big competitors. Worth noting, I was able to dodge the DDOS attacks because I already geo-block everything but EU and US since I am not providing business to those other countries anyways.

    8. Re:Oh no! by h33t+l4x0r · · Score: 5, Informative

      I believe they're the number 2 player after AWS (Amazon Web Services). So a big fish, and it's an impressive accomplishment to give them so much trouble.

    9. Re:Oh no! by Zontar+The+Mindless · · Score: 2

      Exactly. I've never heard of Linode and I run Linux on one of my machines. Is it too much to ask that you give a 1-2 sentence description of why I should care?

      And no, I don't care enough to check wikipedia, nor google. Never heard of 'em, don't care, the summary gives me no reason to change my mind on either.

      Since you seem to have to much free time to troll on the forums and brag about your ignorance, why would anyone bother push information under your nose... learn to use search ..

      Maybe, just maybe, the summary could give those of us who aren't experts on everything under the Sun a reason to care?

      --
      Il n'y a pas de Planet B.
    10. Re:Oh no! by Zontar+The+Mindless · · Score: 2

      KVM (Kernel-based Virtual Machine)

      That is most definitely NOT what KVM stands for. Get off my lawn, etc.

      These folks seem to disagree. Moo?

      --
      Il n'y a pas de Planet B.
    11. Re:Oh no! by vel-ex-tech · · Score: 5, Informative

      Linode is a quite good VPS provider. They have several stock distro installs to choose from (Linux and BSD), and then the sky is the limit. They also pay for user-generated documentation, and the focus is on FLOSS software that you can install and configure on your node. This isn't some PHP MySQL crap. I've been a happy user for years now, running a private mail, web, and IRC server. The prices are quite than reasonable. I'm not sure if they offer Xen nodes anymore since KVM is the way to go.

      My nodes at Fremont haven't been affected yet. Soylentnews, also hosted on Linode, seems to be doing well too.

    12. Re:Oh no! by vel-ex-tech · · Score: 2

      Let me check Fremont.


      > ping xyz.com
      PING xyz.com (555.123.45.67) 56(84) bytes of data.
      64 bytes from lidsfargeg.members.linode.com (555.123.45.67): icmp_seq=1 ttl=50 time=69.0 ms
      64 bytes from lidsfargeg.members.linode.com (555.123.45.67): icmp_seq=2 ttl=50 time=69.0 ms
      64 bytes from lidsfargeg.members.linode.com (555.123.45.67): icmp_seq=3 ttl=50 time=72.8 ms
      64 bytes from lidsfargeg.members.linode.com (555.123.45.67): icmp_seq=4 ttl=50 time=66.7 ms
      64 bytes from lidsfargeg.members.linode.com (555.123.45.67): icmp_seq=5 ttl=50 time=67.2 ms
      64 bytes from lidsfargeg.members.linode.com (555.123.45.67): icmp_seq=6 ttl=50 time=68.7 ms
      64 bytes from lidsfargeg.members.linode.com (555.123.45.67): icmp_seq=7 ttl=50 time=69.0 ms

      Looks good from flyover country. According to the status page, it seems it's primarily the Atlanta datacenter that's affected. Protip: don't spin up all your nodes in the same datacenter! Fortunately, Linode has a few of those. I'm certain they have top women on this, and they'll have this resolved quickly. (Even better if there are strange, up, down, or charmed women on the job. Not sure about the bottom women....)

    13. Re:Oh no! by Vegard · · Score: 2

      DDOS is vandalism, simply. We'd cover it if a mob was throwing rocks at Microsoft, RedHat og IBMs head quarters, would we not? This is just as much vandalism, and there *is* no justification for vandalism.

      This is actually why I am no fan of groups like Anonymous, and do not cheer when they for example hit ISIS targets. There's bound to be a lot of collateral damage, and a lot of innocent people will be targeted.

      As for Linode? God knows what they or some of their customers have done. But this time - like a lot of other innocent people - it actually hit *me* too. I got tired of running a physical machine at home just so I could run all my services, so I got myself a server at Linode. It has a lot better WAF, too.

      So am I satisfied with Linode? You bet. This is not their fault. I am sure they are bound to learn from it, to quicker stop this in the future. But it is not easy, to stop mobs. Because DDOS groups are just that: Mobs! No matter how much you sympatize with a cause, a mob is never justified.

      Period.

    14. Re:Oh no! by i.r.id10t · · Score: 2

      Eh, for individual users and small businesses I'd put them ahead of amazon. As in, live chat with real help, very quick response to emails, proactive support, and for the one period I was playign with AWS much better deal (a micro on AWS was gonna run $15/mo with 8gb disk and 512mb ram, at the time on linode $20 got you 1gb ram and 20gb disk), and now linode has a $10/mo plan. Been a very happy customer of theirs for 10+ years.

      And I've not noticed any issues from the DDOS either. Of course, I run simple mail for my own domains and a few images on a web server, so it isnt like I've got customers trying to get to their stuff....

      --
      Don't blame me, I voted for Kodos
  2. Haven't noticed a thing... by Anonymous Coward · · Score: 3, Informative

    As one of their customers, I haven't noticed a thing. My instances keep on chugging along as if nothing is wrong.

    1. Re:Haven't noticed a thing... by h33t+l4x0r · · Score: 2

      Mine too, mine's in Tokyo, one of the few regions that haven't been hit.

    2. Re:Haven't noticed a thing... by DRichardHipp · · Score: 5, Informative
      https://www.sqlite.org/ is hosted on Linode - has been for over 10 years. The site was off-line for about 10 minutes on Tuesday, but service has been OK otherwise.. The folks at Linode have done a good job of keeping things running. I see now that Chris Aker and his team have had a challenging week.

      I've used a variety of hosting providers, but I always keep coming back to Linode. Their product is competitively priced, they provide exceptional service and support, and they are very simple to use. And, unlike AWS, you don't need a calculator and 2 hours spent parsing fine print in the documentation to figure out how much a given level of service will end up costing you. I highly recommend Linode for your cloud computing needs. I hope they are able to resolve their DDoS problems quickly.

    3. Re:Haven't noticed a thing... by devilspgd · · Score: 4, Interesting

      I've got several Linodes, I've probably seen about 10-15 minutes of downtime total (per node, and not at the same time), so in my case this translated into approximately 8 minutes of customer-facing outage due to my internal redundancy.

      However, my redundancy is within a Linode network, if an entire Linode data-center goes down, so do I, I don't attempt to replicate outside of an individual DC, outside of off-site backups (which I store outside the Linode environment). We do have core infrastructure (DNS, our own mail and system status pages) distributed across multiple providers so that losing a single provider won't take us down, although this is mainly to prevent a situation such as where my Linode account itself is suspended.

      All in all, I'm quite impressed at how well they've handled it.

      --
      Give a man a fish, he'll eat for a day, but teach a man to phish...
    4. Re:Haven't noticed a thing... by srw · · Score: 3, Informative

      I was wondering why siteuptime was reporting brief outages on my Linode. I guess this explains why I've had the worst uptime in years... just got my December report. 99.66% uptime. I've been with Linode for a very long time and can honestly not say a bad word about them. I've really only ever had one bad outage, when the UPSes at HE Dallas blew up. I think that one ended up being about 8 hours. They kept us updated and even set up a webcam so we could watch them working in the datacentre getting things booted back up after power was restored.

  3. Maybe I'm a jerk... by devilspgd · · Score: 4, Interesting

    Okay, I'm probably a jerk, but I don't care and I hope their upstream(s) isn't/aren't helpful.

    I'm a happy Linode customer, but when one of my customers was being targeted by a DDoS extortion scheme which was using a very specific, very blockable attack, Linode's only solution/suggestion was to boot the customer, or wait it out, and in the meantime, they nullrouted my IP. Now I get that nullrouting my IP keeps the rest of the customers in that subnet/node/etc online, but it frustrated me that they wouldn't even attempt to block selectively, and as such, I can't get a ton of sympathy when they're victims of similar attacks.

    And for the record, my customer didn't pay, eventually the DDoS group got bored and moved on.

    --
    Give a man a fish, he'll eat for a day, but teach a man to phish...
    1. Re:Maybe I'm a jerk... by Anonymous Coward · · Score: 3, Interesting

      Well, hopefully you won't mind me sounding like a jerk too, but I'm going to outright call you out as a liar on that one plain and straight.

      As a Linode customer for over a decade now with a good number of nodes still running, and who has previously been the target of DDoS attacks, I have never seen caker or any of their staff respond in such a way to one of their customers.
      They have been nothing but above and beyond helpful with blocking attacks and working with me to keep my (and their own) services up, including custom layer 3 filtering at their upstream providers specifically to address the types of attacks directed at me.

      Yes they null route the specific IPs being attacked during the peek of the attacks to keep services up for their other customers first, and you expecting anything different is not only foolish but demonstrates unrealistic expectations and a failure to grasp how network routing works at the backbone level.

      But to claim they threatened either you or your customer must go has no president, and does not mesh with how they have always handled such things in the past.

      Did you go about threatening them with lawsuits like an entitled prick or something? Maybe then I could see such a response (and not really blame anyone for it)
      But you imply trying to work with them on the problem so I doubt that to be the case, which is why I doubt your explanation of what happened is at all the full story.

      All of that aside however, how fucking petty and low to wish such DDoS attacks on them for no other reason than you being a little frustrated at being offline for a few minutes expecting god like powers from those who clearly don't have them.
      Especially when their entire staff has worked around the clock over the holidays to keep services of entitled pricks like you running instead of being with their friends and families, all over a petty exaggerated grudge.

    2. Re:Maybe I'm a jerk... by mysidia · · Score: 5, Informative

      Null routing an IP address under DDoS attack in an emergency is standard industry practice across all major ISPs and hosting providers; companies that use more advanced techniques either have a few tricks up their sleeves which only work in the most common situations, or they bought some $5 million anti-DoS appliances to help mitigate it (usually).

      The simple fact is DoS mitigation is not part of a basic hosting service, once an attack exceeds a few million packets per second, or a couple Gigabits: you are simply not paying network providers enough money for it to be feasible for any ISP to come close to justifying effective DoS mitigation for those rare sizes of attack, for every customer, because the cost involves provisioning hundreds of million$$$ in extra upstream capacity, internal network capacity, and operations staff.

      Then even with all that extra capital spend: (1) It's still not possible to make every attack seamless, Null-routing might still be required in cases, there will still be outages, people like the above will still be unhappy, And.... (2) Most ISPs don't have that much throwaway cash, and most hosting customers aren't going to be willing to pay their share of what it costs to provision 10000x as much capacity as needed.

      (3) Its less expensive to just shed overly-demanding customers who pay little by allowing them to make themselves unhappy and go to a competitor. If someone's paying $100 a month and their site is constantly getting DDoS'd, then it makes perfect sense to terminate them as a customer to, and let the other 10000 $100/Month customers have a better experience, instead of leaving due to the DoS being suffered as a result of 1 customer.

      And if someone wants to arrange for their website to be handled differently, then this is part of a negotiation that should be made with the ISP or provider before turning up hosting service and added to the contract, with response SLA and recourse/refund policy.

      Or you're better off enlisting a 3rd party DoS-scrubbing service such as CloudFlare to conceal your infrastructure from attackers.

      There are also DoS-cleanup services that work at a network range level where your DDoS provider announces your /24 into BGP, cleans DoS, and forwards you traffic.

      Many ISPs do have the flexibility for alternate handling of DDoS, up to a certain point, they can avoid Null-routing an IP, or avoid the Null-routing of one IP from making your service unavailable.... generally, the cost will be much higher --- E.g. $10,000 per month instead of $100 per month.

      Forget about attempting to negotiate expert-level DoS management that will require the provisioning of engineer and infrastructure resources in advance that are quite costly to the providers to keep on hand, Unless you are willing to pay sufficiently to be a large client of the provider with a multi-year committed contract and cover the costs of those extra resources plus sizable profit.

      Also: to host a website resiliently, however, the provider will most likely require that the website be served from multiple server farms in multiple IP ranges with an anycasted internet presence for both the services' IP addresses, and the supporting DNS services.

      This is because in spite of additional resources, it might still be necessary at times to fall back to Null-routing.

    3. Re:Maybe I'm a jerk... by devilspgd · · Score: 3, Interesting

      I see why you're posting as A/C. Try reading again: Linode didn't threaten me or the customer or anyone else, we (my customer, and me as a host) were threatened by a DDoS extortion scheme -- A "Nice web site you host there, it would be a shame if someone were to keep up a sustained DDoS against it"

      And no, I didn't threaten a lawsuit or anything else, I just asked them for information on the type of attack, and later once I identified the type of attack, help filtering rather than a complete nullroute.

      They were polite, but completely unhelpful in terms of even providing any information about the particulars of the attack (one source or multiple, port numbers, type of traffic, etc) "Luckily" the DDoS hit servers I run on another network too, and the network operator there was able to provide me with said details, which helped to mitigate the attacks.

      --
      Give a man a fish, he'll eat for a day, but teach a man to phish...
    4. Re:Maybe I'm a jerk... by devilspgd · · Score: 4, Informative

      I didn't wish such DDoS attacks on them at all. Has /. reading comprehension really fallen this low?

      What I hope is that their provider is as unhelpful to them as Linode was to me when I was a victim of similar, ongoing and sustained attacks, as it will help them understand the difficulty that customers face and that they're left struggling to resolve it on their own because if so, they may develop both sympathy and tools that can be used to protect both themselves and their customers in the future.

      If "Oh, just shut everything down and wait it out" is good enough for me, it should be good enough for them. If not, well, maybe they'll improve after having a bit more personal experience being the victim.

      And for the record, I'm still a Linode customer (and have more services with them now than I did then); I was just disappointed at their lack of usefulness.

      --
      Give a man a fish, he'll eat for a day, but teach a man to phish...
    5. Re:Maybe I'm a jerk... by ThatsMyNick · · Score: 2

      It is simple, you dont pay them enough to do that for you. They provide cheap, cookie cutter setups. If the setup is not for you, they do not do custom setups, they dont take odd (but possibly simple) requests. Find another business, really.

  4. Damnit!!! by FudRucker · · Score: 2

    this is why we cant have nice things!!!

    --
    Politics is Treachery, Religion is Brainwashing
  5. Re:Why? by mysidia · · Score: 3, Insightful

    These attacks cost money. People don't do them for fun anymore.

    You are sadly mistaken.... just b/c attacks cost money, does not mean people do not still launch them willy-nilly ---- you are a hosting provider, and some website you are hosting makes the wrong person or wrong group angry, or they have a political message to send, they may attack the entire provider.

    Although, quite often such large-scale attacks are launched, then followed up by "ransom demands", and as long as the target does not pay the huge cash ransom, the attacks continue, And the attacker stands to make much more $$$ from ransom than the attacks would have cost them.

  6. Re:Why? by runningduck · · Score: 2

    Linode is a virtual server provider. Easy as pie to manage your service starting new instances and deleting instances through their control panel.

    --
    -rd
  7. Re:Why? by runningduck · · Score: 2

    "If you think the universe is big, you should see the source code."

    --
    -rd
  8. BBC often down too: related by Maow · · Score: 2

    I've got to wonder if BBC's issues are related to Linode being hit, or if BBC is the target and Linode is suffering for it?

    I've seen BBC have issues in the past, but never as bad and for as long as since Christmas.

    As recently as a minute ago I couldn't get a page to load.

  9. Re: Don't take my word for it by ihtoit · · Score: 2

    I dunno, ten bucks a month for more power than a four year old laptop and a 40Gig pipe and guaranteed three-nines uptime is pretty fucking good.

    --
    Political debates have me rolling my eyes so much I think I got optical whiplash. I should sue. - Foamy The Squirrel