Amid Major Internet Outages, Affected Websites Have Lessons To Learn (zdnet.com)

Earlier today, Dyn, an internet infrastructure company, was hit by several DDoS attacks, which interestingly affected several popular websites including The New York Times, Reddit, Spotify, and Twitter that were directly or indirectly using Dyn's services. The attack is mostly visible across the US eastern seaboard with rest of the world noticing a few things broken here and there. Dyn says it's currently investigating a second round of DDoS attacks, though the severity of the outage is understandably less now. In the meantime, the Homeland Security said that it is aware of the attack and is investigating "all potential causes." Much of who is behind these attacks is unknown for now, and it is unlikely that we will know all the details until at least a few days. The attacks however have revealed how unprepared many websites are when their primary DNS provider goes down. ZDNet adds: The elephant in the room is that this probably shouldn't have happened. At very least there's a lot to learn already about the frailty of the internet DNS system, and the lack of failsafes and backups for websites and tech companies that rely on outsourced DNS service providers. "It's also a reminder of one risk of relying on multi-tenant service providers, be they DNS, or a variety of many other managed cloud service providers," said Steve Grobman, chief technology officer at Intel Security. Grobman warned that because this attack worked, it can be exploited again. "Given how much of our connected world must increasingly rely upon such cloud service providers, we should expect more such disruptions," he said. "We must place a premium of service providers that can present backup, failover, and enhance security capabilities allowing them to sustain and deflect such attacks." And that's key, because even though Dyn is under attack, it's the sites and services that rely on its infrastructure who should rethink their own "in case of emergency" failsafes. It may only be the east coast affected but lost traffic means lost revenue. Carl Levine, senior technical evangelist for NS1, another major managed DNS provider, said that the size and scale of recent attacks "has far exceeded what the industry thought was the upper end of the spectrum." "Large companies need to constantly upgrade their flood defenses. Some approaches that worked just a few years ago are now basically useless," said Kevin Curran, senior member with IEEE.We also recommend reading security reporter Brian Krebs's take on this.

Comments

First lesson

[unixisc]

Make your website IPv6 only, so that DDOS attacks would have to be totally re-engineered to target them, and that too will be a tall order.

Re:First lesson

[m.dillon]

I have two major beefs with IPV6. The first is that the end-point 2^48 switch address space wasn't well thought-through. Hey, wouldn't it be great if we didn't have to use NAT and give all of those IOT devices their own IPV6 address? Well... no actually, NAT does a pretty good job of obscuring the internal topology of the end-point network. Just having a statefull firewall and no NAT exposes the internal topology. Not such a good idea.

The second is that all the discovery protocols were left unencrypted and made complex enough to virtually guarantee a plethora of possible exploits. Some have been discovered and fixed, I guarantee there are many more in the wings. IPV4 security is a well known problem with well known solutions. IPV6 security is a different beast entirely.

Other problems including the excessively flexible protocol layering allowing for all sorts of encapsulation tricks (some of which have already been demonstrated), pasting on a 'mandatory' IPSEC without integration with a mandatory secure validation framework (making it worthless w/regards to generic applications being able to assert a packet-level secure connection), assumptions that the address space would be too big to scan (yah right... the hackers didn't get that memo my tcpdump tells me), not making use of MAC-layer features that would have improved local LAN security, if only a little. Also idiotically and arbitrarily blocking off a switch subspace, eating 48 bits for no good reason and trying to disallow routing within that space (which will soon have to be changed considering that number of people who want to have stateful *routers* to break up their sub-48-bit traffic and who have no desire whatsoever to treat those 48 bits as one big switched sub-space).

The list goes on. But now we are saddled with this pile, so we have to deal with it.

-Matt

Re:First lesson

[MightyMartian]

NAT may do a good job obscuring internal topology, but it does saw at considerable cost; breaking the end to end concept of the original ARPANET structure, requiring more resources, and creating far greater complexity for routers. Yes, a flat address space that sits in the public address space might, on the surface, expose more devices, but this is where firewalls come into play. I can still have a rather complex topology, but now I have to worry less about routing and connection tables, and can use less resource expensive techniques like tagging.

It was never IPv6's intention to be more secure, and you're right that many existing issues will remain with IPv6, and there will likely be new ones, but one thing is certain, if the solution is NAT, then that solution is worse than the disease it purports to cure. And it isn't as if NAT can't be vulnerable in its own way, and the only way to make it less vulnerable is, you guessed it, firewalls, authentication, and other security measures which are also needed in an IPv6 world.

What's the Solution?

[BenFranske]

I've heard a lot of people today saying there's a problem. Several of the commenters (on Brian Krebs' blog for example, on the NANOG list for another, and probably soon here on ./) say we should do something to fix this so it doesn't happen again. What I haven't heard is a real proposal about what to do about stopping DDoS attacks.

Re:What's the Solution?

[Anonymous Coward]

I've seen it a million times, in no small part because I've posted it myself:

ISPs need to start egress filtering to block spoofed packets coming from end users with forged source addresses. If a packet comes from joe blow's cable modem with a source IP from some other country, it should just be dropped.

Flood defenses?

[m.dillon]

There is no flood defense possible for most businesses at the tail-end of the pipe. When an attacker pushes a terrabit/s at you and at all the routers in the path leading to you as well as other leafs that terminate at those routers, from 3 million different IP addresses from compromised IOT devices, your internet pipes are dead, no matter how much redundancy you have.

Only the biggest companies out there can handle these kinds of attacks. The backbone providers have some defenses, but it isn't as simple as just blocking a few IPs.

-Matt

Re:Flood defenses?

[Dadoo]

but it isn't as simple as just blocking a few IPs.

And this is why people need to be fined, if a device on their home network is found to be part of a botnet. Individuals need to be responsible for their networks, because the authorities are virtually powerless against botnets, Unless it costs them money, people just won't care.

Re:Flood defenses?

[0123456]

And then what?

They buy a device that's horribly insecure, the manufacturer sends out one security update, then abandons it, and it becomes part of a botnet. And you fine the person who bought it?

Actually, you're right. That's a great idea, because it will kill the whole 'Internet of Things' idiocy overnight. No-one will risk attaching anything to their network if they can't verify it's secure.

Tackling Mirai

[subk]

Now that the source code for Mirai is out there being used, is there something that can be done to tackle the spread? Call me crazy, but perhaps a modified version could go out and actually change the passwords on these insecure IoT devices to random strings? Sure, the owner would lose access to the device.. But it would alert them that something was wrong, and stop the spread of Mirai.

Is it really a war?

[beheaderaswp]

I've been looking at the mainstream media outlets and they are reporting on this attack as if we were just invaded by Russia.

This was an attack against DNS... at worst this type of attack stops people from "doing something". That "something" could be playing Pokemon... or banking... or working. But it doesn't "take down" the internet.

The internet is just fine. To take down the "whole internet" you'd have to attack routers. And the numbers of routers exceed the ability of anyone to saturate them. So why does the media get all hyped up when Twitter goes down?

It irks me so badly that the media and the general public get so completely flustered when some third world country, or a group of kids, decide to play games with the system. And that is all it is.

Certainly we should defend against disruptions like this. How they are done should be researched. Perhaps in the future the system can be hardened so it's incredibly difficult to attack it.

But it's a pretty minor league attack against the "internet". Twitter is down? The NYT?

I just turned 50 last year. Still up to date on tech. Still as sharp as I was at 25 when I lugged a Compaq suitcase around. This seems like such a small issue to me. When the real issue should be router security, the idiotic idea of tying SSL certs to domain names, or the sad security of home routers.

Yet more proof and confirmation that...

[X86BSD]

CLOUD anything and outsourcing your infrastructure because you are lazy and/or cheap is a BAD IDEA. Consolidating services you no longer control to a third party means you've lost the ability to survive these attacks.

November 1987, RFC 1034. Secondary DNS servers

[raymorris]

For this specific attack, set up a secondary name server, using a secondary provider.

In November 1987, RFC 1034 was published. It describes how secondary DNS servers automatically sync from the primary. For about twelve years, people took that seriously. The used ar least two name servers that were unlikely to be affected by the same problem - separated geographically far apart and using two (or more) different network providers. Nowadays it's likely their two name servers are sitting right on top of each other in the same rack.

If both your DNS servers are with the same provider, wherher that be Amazon, DynDNS, or any other single provider, they are subject to fail due to the same cause, at the same time.

Btw ona different, but related topic - there's also an RFC for exactly how to build CDNs (reverse proxies) that actually work right. We've known how to do that correctly for decades, so everybody can read the damn RFC and stop inventing new ways to completely screw it up. First hint - the protocol for reverse proxies has been around far longer than the buzzword "CDN" that's now used to sell them.