Linode Under DDoS Since Christmas (linode.com)

hol writes: Linode has been getting hit with DDoS attacks since Christmas Day, and it looks like their pain is set to continue. The attackers are rotating DDoS traffic through various regions of Linode's service. They say, "All of these attacks have occurred multiple times. Over the course of the last week, we have seen over 30 attacks of significant duration and impact. As we have found ways to mitigate these attacks, the vectors used inevitably change. As of this afternoon, we have mostly hardened ourselves against the above attack vectors, but we expect more to come. ... Once these attacks stop, we plan to share a complete technical explanation about what has been happening." See their status page for updates.

Comments

Oh no!

[Anonymous Coward]

WTF is "Linode"?

Re:Oh no!

[ArchieBunker]

The janitors who run this site can't even be bothered to hyperlink Linode to Wikipedia.

Re:Oh no!

[Snotnose]

Exactly. I've never heard of Linode and I run Linux on one of my machines. Is it too much to ask that you give a 1-2 sentence description of why I should care?

And no, I don't care enough to check wikipedia, nor google. Never heard of 'em, don't care, the summary gives me no reason to change my mind on either.

Re:Oh no!

[Razed By TV]

This is becoming a regular problem on /. Article titles and summaries are increasingly assuming that people have the obscure knowledge of the topic to actually care. In this case, who Linode is, what makes Linode important, why this DDoS merits more attention than other attacks, etc.

It used to be that when I saw a title/summary that I was unfamiliar with, I could follow it and expect to learn something from it.
Now I find out that JustAnotherCompany experienced JustAnotherThingThatHappensOnTheInternet.

I googled Linode, so I guess I learned something. Cloud hosting/virtual servers. Are they big fish, little fish, do they host someone big, are they known for something they did in particular? Well, I have better things to do than research it.

Re:Oh no!

[h33t l4x0r]

I believe they're the number 2 player after AWS (Amazon Web Services). So a big fish, and it's an impressive accomplishment to give them so much trouble.

Re:Oh no!

[vel-ex-tech]

Linode is a quite good VPS provider. They have several stock distro installs to choose from (Linux and BSD), and then the sky is the limit. They also pay for user-generated documentation, and the focus is on FLOSS software that you can install and configure on your node. This isn't some PHP MySQL crap. I've been a happy user for years now, running a private mail, web, and IRC server. The prices are quite than reasonable. I'm not sure if they offer Xen nodes anymore since KVM is the way to go.

My nodes at Fremont haven't been affected yet. Soylentnews, also hosted on Linode, seems to be doing well too.

Maybe I'm a jerk...

[devilspgd]

Okay, I'm probably a jerk, but I don't care and I hope their upstream(s) isn't/aren't helpful.

I'm a happy Linode customer, but when one of my customers was being targeted by a DDoS extortion scheme which was using a very specific, very blockable attack, Linode's only solution/suggestion was to boot the customer, or wait it out, and in the meantime, they nullrouted my IP. Now I get that nullrouting my IP keeps the rest of the customers in that subnet/node/etc online, but it frustrated me that they wouldn't even attempt to block selectively, and as such, I can't get a ton of sympathy when they're victims of similar attacks.

And for the record, my customer didn't pay, eventually the DDoS group got bored and moved on.

Re:Maybe I'm a jerk...

[mysidia]

Null routing an IP address under DDoS attack in an emergency is standard industry practice across all major ISPs and hosting providers; companies that use more advanced techniques either have a few tricks up their sleeves which only work in the most common situations, or they bought some $5 million anti-DoS appliances to help mitigate it (usually).

The simple fact is DoS mitigation is not part of a basic hosting service, once an attack exceeds a few million packets per second, or a couple Gigabits: you are simply not paying network providers enough money for it to be feasible for any ISP to come close to justifying effective DoS mitigation for those rare sizes of attack, for every customer, because the cost involves provisioning hundreds of million$$$ in extra upstream capacity, internal network capacity, and operations staff.

Then even with all that extra capital spend: (1) It's still not possible to make every attack seamless, Null-routing might still be required in cases, there will still be outages, people like the above will still be unhappy, And.... (2) Most ISPs don't have that much throwaway cash, and most hosting customers aren't going to be willing to pay their share of what it costs to provision 10000x as much capacity as needed.

(3) Its less expensive to just shed overly-demanding customers who pay little by allowing them to make themselves unhappy and go to a competitor. If someone's paying $100 a month and their site is constantly getting DDoS'd, then it makes perfect sense to terminate them as a customer to, and let the other 10000 $100/Month customers have a better experience, instead of leaving due to the DoS being suffered as a result of 1 customer.

And if someone wants to arrange for their website to be handled differently, then this is part of a negotiation that should be made with the ISP or provider before turning up hosting service and added to the contract, with response SLA and recourse/refund policy.

Or you're better off enlisting a 3rd party DoS-scrubbing service such as CloudFlare to conceal your infrastructure from attackers.

There are also DoS-cleanup services that work at a network range level where your DDoS provider announces your /24 into BGP, cleans DoS, and forwards you traffic.

Many ISPs do have the flexibility for alternate handling of DDoS, up to a certain point, they can avoid Null-routing an IP, or avoid the Null-routing of one IP from making your service unavailable.... generally, the cost will be much higher --- E.g. $10,000 per month instead of $100 per month.

Forget about attempting to negotiate expert-level DoS management that will require the provisioning of engineer and infrastructure resources in advance that are quite costly to the providers to keep on hand, Unless you are willing to pay sufficiently to be a large client of the provider with a multi-year committed contract and cover the costs of those extra resources plus sizable profit.

Also: to host a website resiliently, however, the provider will most likely require that the website be served from multiple server farms in multiple IP ranges with an anycasted internet presence for both the services' IP addresses, and the supporting DNS services.

This is because in spite of additional resources, it might still be necessary at times to fall back to Null-routing.

Re:Maybe I'm a jerk...

[devilspgd]

I didn't wish such DDoS attacks on them at all. Has /. reading comprehension really fallen this low?

What I hope is that their provider is as unhelpful to them as Linode was to me when I was a victim of similar, ongoing and sustained attacks, as it will help them understand the difficulty that customers face and that they're left struggling to resolve it on their own because if so, they may develop both sympathy and tools that can be used to protect both themselves and their customers in the future.

If "Oh, just shut everything down and wait it out" is good enough for me, it should be good enough for them. If not, well, maybe they'll improve after having a bit more personal experience being the victim.

And for the record, I'm still a Linode customer (and have more services with them now than I did then); I was just disappointed at their lack of usefulness.

Re:Haven't noticed a thing...

[DRichardHipp]

https://www.sqlite.org/ is hosted on Linode - has been for over 10 years. The site was off-line for about 10 minutes on Tuesday, but service has been OK otherwise.. The folks at Linode have done a good job of keeping things running. I see now that Chris Aker and his team have had a challenging week.

I've used a variety of hosting providers, but I always keep coming back to Linode. Their product is competitively priced, they provide exceptional service and support, and they are very simple to use. And, unlike AWS, you don't need a calculator and 2 hours spent parsing fine print in the documentation to figure out how much a given level of service will end up costing you. I highly recommend Linode for your cloud computing needs. I hope they are able to resolve their DDoS problems quickly.

Re:Haven't noticed a thing...

[devilspgd]

I've got several Linodes, I've probably seen about 10-15 minutes of downtime total (per node, and not at the same time), so in my case this translated into approximately 8 minutes of customer-facing outage due to my internal redundancy.

However, my redundancy is within a Linode network, if an entire Linode data-center goes down, so do I, I don't attempt to replicate outside of an individual DC, outside of off-site backups (which I store outside the Linode environment). We do have core infrastructure (DNS, our own mail and system status pages) distributed across multiple providers so that losing a single provider won't take us down, although this is mainly to prevent a situation such as where my Linode account itself is suspended.

All in all, I'm quite impressed at how well they've handled it.