Amid Major Internet Outages, Affected Websites Have Lessons To Learn (zdnet.com)

Earlier today, Dyn, an internet infrastructure company, was hit by several DDoS attacks, which interestingly affected several popular websites including The New York Times, Reddit, Spotify, and Twitter that were directly or indirectly using Dyn's services. The attack is mostly visible across the US eastern seaboard with rest of the world noticing a few things broken here and there. Dyn says it's currently investigating a second round of DDoS attacks, though the severity of the outage is understandably less now. In the meantime, the Homeland Security said that it is aware of the attack and is investigating "all potential causes." Much of who is behind these attacks is unknown for now, and it is unlikely that we will know all the details until at least a few days. The attacks however have revealed how unprepared many websites are when their primary DNS provider goes down. ZDNet adds: The elephant in the room is that this probably shouldn't have happened. At very least there's a lot to learn already about the frailty of the internet DNS system, and the lack of failsafes and backups for websites and tech companies that rely on outsourced DNS service providers. "It's also a reminder of one risk of relying on multi-tenant service providers, be they DNS, or a variety of many other managed cloud service providers," said Steve Grobman, chief technology officer at Intel Security. Grobman warned that because this attack worked, it can be exploited again. "Given how much of our connected world must increasingly rely upon such cloud service providers, we should expect more such disruptions," he said. "We must place a premium of service providers that can present backup, failover, and enhance security capabilities allowing them to sustain and deflect such attacks." And that's key, because even though Dyn is under attack, it's the sites and services that rely on its infrastructure who should rethink their own "in case of emergency" failsafes. It may only be the east coast affected but lost traffic means lost revenue. Carl Levine, senior technical evangelist for NS1, another major managed DNS provider, said that the size and scale of recent attacks "has far exceeded what the industry thought was the upper end of the spectrum." "Large companies need to constantly upgrade their flood defenses. Some approaches that worked just a few years ago are now basically useless," said Kevin Curran, senior member with IEEE.We also recommend reading security reporter Brian Krebs's take on this.

Comments

First lesson

[unixisc]

Make your website IPv6 only, so that DDOS attacks would have to be totally re-engineered to target them, and that too will be a tall order.

Re:First lesson

[darkain]

How exactly would being on a /64 prevent such an attack against a publicly facing entity? These attacks are not address space scanning attacks at all, they are known and publicly published IP addresses (in this case, DNS servers). Flood the public facing IP (the DNS server) would be exactly the same if IPv4 or IPv6. The only thing this would temporarily mitigate is the fact there are far fewer devices/users on the IPv6 network, so less of a botnet to control currently.

Re: First lesson

[dilvish_the_damned]

If your app can find the right server for the service, so can the attacking software.

Re:First lesson

[m.dillon]

I have two major beefs with IPV6. The first is that the end-point 2^48 switch address space wasn't well thought-through. Hey, wouldn't it be great if we didn't have to use NAT and give all of those IOT devices their own IPV6 address? Well... no actually, NAT does a pretty good job of obscuring the internal topology of the end-point network. Just having a statefull firewall and no NAT exposes the internal topology. Not such a good idea.

The second is that all the discovery protocols were left unencrypted and made complex enough to virtually guarantee a plethora of possible exploits. Some have been discovered and fixed, I guarantee there are many more in the wings. IPV4 security is a well known problem with well known solutions. IPV6 security is a different beast entirely.

Other problems including the excessively flexible protocol layering allowing for all sorts of encapsulation tricks (some of which have already been demonstrated), pasting on a 'mandatory' IPSEC without integration with a mandatory secure validation framework (making it worthless w/regards to generic applications being able to assert a packet-level secure connection), assumptions that the address space would be too big to scan (yah right... the hackers didn't get that memo my tcpdump tells me), not making use of MAC-layer features that would have improved local LAN security, if only a little. Also idiotically and arbitrarily blocking off a switch subspace, eating 48 bits for no good reason and trying to disallow routing within that space (which will soon have to be changed considering that number of people who want to have stateful *routers* to break up their sub-48-bit traffic and who have no desire whatsoever to treat those 48 bits as one big switched sub-space).

The list goes on. But now we are saddled with this pile, so we have to deal with it.

-Matt

Re:First lesson

[MightyMartian]

NAT may do a good job obscuring internal topology, but it does saw at considerable cost; breaking the end to end concept of the original ARPANET structure, requiring more resources, and creating far greater complexity for routers. Yes, a flat address space that sits in the public address space might, on the surface, expose more devices, but this is where firewalls come into play. I can still have a rather complex topology, but now I have to worry less about routing and connection tables, and can use less resource expensive techniques like tagging.

It was never IPv6's intention to be more secure, and you're right that many existing issues will remain with IPv6, and there will likely be new ones, but one thing is certain, if the solution is NAT, then that solution is worse than the disease it purports to cure. And it isn't as if NAT can't be vulnerable in its own way, and the only way to make it less vulnerable is, you guessed it, firewalls, authentication, and other security measures which are also needed in an IPv6 world.

Re:First lesson

[sabri]

go back to the whiteboard

APK was right all along! C:\WINDOWS\HOSTS is the solution ;)

*goes to hide under a rock*

Re:First lesson

[JustAnotherOldGuy]

Except that in reality the way it works...

Except that in reality some ISPs are owned by the Russian Business Network (RBN), and they'll be given 100 million IPs to play with, and then another 100 million, and so on. The RBN owns lots of ISPs that are known for their friendliness towards "bulletproof" hosting companies and for working hand-in-hand with spammers.

-

The spammer would either need a new block of addresses from the ISP or a new ISP, effectively the same situation you have now with IPv4.

No, it's not the same situation because the address space that will be available to these criminal ISPs will be magnitudes of order larger than with IPV4. An ISP now may have a hundred thousand IPs to allocate (if that many), but now they'll have tens or hundreds of millions.

Seriously, this is going to be a problem, and more than a few security professionals have been discussing this problem for a while now.

What's the Solution?

[BenFranske]

I've heard a lot of people today saying there's a problem. Several of the commenters (on Brian Krebs' blog for example, on the NANOG list for another, and probably soon here on ./) say we should do something to fix this so it doesn't happen again. What I haven't heard is a real proposal about what to do about stopping DDoS attacks.

Re:What's the Solution?

[Anonymous Coward]

I've seen it a million times, in no small part because I've posted it myself:

ISPs need to start egress filtering to block spoofed packets coming from end users with forged source addresses. If a packet comes from joe blow's cable modem with a source IP from some other country, it should just be dropped.

Re: What's the Solution?

[BenFranske]

1) Yes, poorly designed IoT devices make the problem worse but it's existed long before IoT came along. 2) What qualifies as an IoT device, every Arduino with an Ethernet/WiFi port? The code isn't on them until you program them... 3) If mass regulation of all network connected products is the only way we have a problem because you're never going to get global agreement on that and it's going to be nearly impossible to enforce.

Re:What's the Solution?

[guruevi]

A) Avoid a single point of failure (the cause of downtime across all these providers)
B) avoid using a single point of failure
C) stop using public DNS (or DNS at all) for self-configuration and discovery of your hosted servers
D) stop using a single provider for all your stuff

Re:What's the Solution?

[guruevi]

Not how the Internet works. Yes that's true on the edges but once you enter into the public Internet, packets could be routed from anywhere to anywhere. The only solution here is to shut down ISPs that are participants but you're talking about getting participation from people that often are themselves involved in the criminal enterprise (that's true for US, Europese, Chinese etc providers) and are profiting from these attacks through overage fees etc.

You wouldn't imagine but even providers like Verizon won't shut off mobile connections because they are often charging their customers per GB consumer. A lot of sleazy hosting provider (the cheap $5/mo.VPS) simply delays intentionally or unintentionally because they don't have the staff to keep up and they are often paid for by the criminals.

Re:What's the Solution?

[Archangel Michael]

It is worse than that.The problem with DDOS is that the real victim probably doesn't know about it.

The proper way to thwart these kinds of attacks is to have a method of detecting them and then cutting off people who are making an inordinate amount of those kinds of packets aimed at that address. The solution to a coordinated attack is a coordinated response.

Look at the Bright Side

[tmjva]

At least the hackers didn't bother shutting down Slashd...

Re:Look at the Bright Side

[MachineShedFred]

Well, we all know that Slashdot uses the mighty APK HOSTS engine to protect it!

Just hang around here long enough, you'll see everything you've ever wanted to know about it.

Flood defenses?

[m.dillon]

There is no flood defense possible for most businesses at the tail-end of the pipe. When an attacker pushes a terrabit/s at you and at all the routers in the path leading to you as well as other leafs that terminate at those routers, from 3 million different IP addresses from compromised IOT devices, your internet pipes are dead, no matter how much redundancy you have.

Only the biggest companies out there can handle these kinds of attacks. The backbone providers have some defenses, but it isn't as simple as just blocking a few IPs.

-Matt

Re:Flood defenses?

[Dadoo]

but it isn't as simple as just blocking a few IPs.

And this is why people need to be fined, if a device on their home network is found to be part of a botnet. Individuals need to be responsible for their networks, because the authorities are virtually powerless against botnets, Unless it costs them money, people just won't care.

Re:Flood defenses?

[0123456]

And then what?

They buy a device that's horribly insecure, the manufacturer sends out one security update, then abandons it, and it becomes part of a botnet. And you fine the person who bought it?

Actually, you're right. That's a great idea, because it will kill the whole 'Internet of Things' idiocy overnight. No-one will risk attaching anything to their network if they can't verify it's secure.

Tackling Mirai

[subk]

Now that the source code for Mirai is out there being used, is there something that can be done to tackle the spread? Call me crazy, but perhaps a modified version could go out and actually change the passwords on these insecure IoT devices to random strings? Sure, the owner would lose access to the device.. But it would alert them that something was wrong, and stop the spread of Mirai.

dare I say it? APK APK APK!

[Anonymous Coward]

As a backup everybody should maintain a HOSTS file with every internet address ever in it.
Can't DDOS that.

Lameness filter encountered. Post aborted! Filter error: Please use fewer 'junk' characters.

Proof the system is rigged...

Re:DNS Replication Service Suggestions?

[Qzukk]

How about http://dyn.com/secondary-dns/ ?

This is a test. This is ONLY a test.

[cfalcon]

If this had been an actual attack, all internet services would be rendered inoperative for long enough for whomever the fuck is doing this to have accomplished whatever the fuck awfulness they desire.

Re:dare I say it? APK APK APK!

[barbariccow]

HOSTS file

sshhh... If you say H***** file 3 times in a mirror you'll summon APK... DO NOT ATTEMPT!

Is it really a war?

[beheaderaswp]

I've been looking at the mainstream media outlets and they are reporting on this attack as if we were just invaded by Russia.

This was an attack against DNS... at worst this type of attack stops people from "doing something". That "something" could be playing Pokemon... or banking... or working. But it doesn't "take down" the internet.

The internet is just fine. To take down the "whole internet" you'd have to attack routers. And the numbers of routers exceed the ability of anyone to saturate them. So why does the media get all hyped up when Twitter goes down?

It irks me so badly that the media and the general public get so completely flustered when some third world country, or a group of kids, decide to play games with the system. And that is all it is.

Certainly we should defend against disruptions like this. How they are done should be researched. Perhaps in the future the system can be hardened so it's incredibly difficult to attack it.

But it's a pretty minor league attack against the "internet". Twitter is down? The NYT?

I just turned 50 last year. Still up to date on tech. Still as sharp as I was at 25 when I lugged a Compaq suitcase around. This seems like such a small issue to me. When the real issue should be router security, the idiotic idea of tying SSL certs to domain names, or the sad security of home routers.

Yet more proof and confirmation that...

[X86BSD]

CLOUD anything and outsourcing your infrastructure because you are lazy and/or cheap is a BAD IDEA. Consolidating services you no longer control to a third party means you've lost the ability to survive these attacks.

November 1987, RFC 1034. Secondary DNS servers

[raymorris]

For this specific attack, set up a secondary name server, using a secondary provider.

In November 1987, RFC 1034 was published. It describes how secondary DNS servers automatically sync from the primary. For about twelve years, people took that seriously. The used ar least two name servers that were unlikely to be affected by the same problem - separated geographically far apart and using two (or more) different network providers. Nowadays it's likely their two name servers are sitting right on top of each other in the same rack.

If both your DNS servers are with the same provider, wherher that be Amazon, DynDNS, or any other single provider, they are subject to fail due to the same cause, at the same time.

Btw ona different, but related topic - there's also an RFC for exactly how to build CDNs (reverse proxies) that actually work right. We've known how to do that correctly for decades, so everybody can read the damn RFC and stop inventing new ways to completely screw it up. First hint - the protocol for reverse proxies has been around far longer than the buzzword "CDN" that's now used to sell them.

Real Business Implications of Internet 3

[WillAffleckUW]

Look, when we built the Internet (back in the ARPA days), it was restricted to trusted players at military and research universities.

Then we let in the unwashed masses.

Then some morons decided to give Internet capability to every single device in the Internet of Things.

First principles, people:

Build one Internet based on IPv6sec for the trusted peers. The backbone.

Build a second Internet for the identified non-object computers based on IPv6sec. The unwashed masses. If parts misbehave, turn off their feeds until they fix them. Drought solves lots of problems.

Build a third Internet for the Internet of Things based on IPv6 and IPv4. Restrict the ports and traffic to essentials. So you can't play Disney in your car, too bad.

Solution?

[mrPalomar]

Is it time for blockchain DNS?

Re:Solution?

[guruevi]

No, just DNS the way it was intended. DNS and all early Internet services were designed to withstand nuclear war and attacks by state-sized actors, actually specifically designed to withstand an attack from Russia.

The problem is the cloud has aggregated all that diversity of everyone running their own services into a handful of really big corporations. Today's just a reminder that any one of those corporations has a significant amount of control if it were a truly bad actor. Imagine Dyn intentionally pointing all the Twitter etc DNS records elsewhere, they did it for their "free" accounts a decade ago just to make them pay.

It seems no one at those big corporations remembers the true history of DynDNS, and how they screwed their customers over. I was surprised they were still in business at all.

OpenDNS SmartCache

[Jayfar]

I was reading elsewhere that users utilizing OpenDNS' SmartCache feature were unaffected. Basically, in the event that a domain's authoritative servers all become unavailable, smartcache uses the last known good resource records, regardless of whether their TTL has expired. Are any of the other DNS providers and ISPs utilizing anything similar?