Amid Major Internet Outages, Affected Websites Have Lessons To Learn (zdnet.com)

Earlier today, Dyn, an internet infrastructure company, was hit by several DDoS attacks, which interestingly affected several popular websites including The New York Times, Reddit, Spotify, and Twitter that were directly or indirectly using Dyn's services. The attack is mostly visible across the US eastern seaboard with rest of the world noticing a few things broken here and there. Dyn says it's currently investigating a second round of DDoS attacks, though the severity of the outage is understandably less now. In the meantime, the Homeland Security said that it is aware of the attack and is investigating "all potential causes." Much of who is behind these attacks is unknown for now, and it is unlikely that we will know all the details until at least a few days. The attacks however have revealed how unprepared many websites are when their primary DNS provider goes down. ZDNet adds: The elephant in the room is that this probably shouldn't have happened. At very least there's a lot to learn already about the frailty of the internet DNS system, and the lack of failsafes and backups for websites and tech companies that rely on outsourced DNS service providers. "It's also a reminder of one risk of relying on multi-tenant service providers, be they DNS, or a variety of many other managed cloud service providers," said Steve Grobman, chief technology officer at Intel Security. Grobman warned that because this attack worked, it can be exploited again. "Given how much of our connected world must increasingly rely upon such cloud service providers, we should expect more such disruptions," he said. "We must place a premium of service providers that can present backup, failover, and enhance security capabilities allowing them to sustain and deflect such attacks." And that's key, because even though Dyn is under attack, it's the sites and services that rely on its infrastructure who should rethink their own "in case of emergency" failsafes. It may only be the east coast affected but lost traffic means lost revenue. Carl Levine, senior technical evangelist for NS1, another major managed DNS provider, said that the size and scale of recent attacks "has far exceeded what the industry thought was the upper end of the spectrum." "Large companies need to constantly upgrade their flood defenses. Some approaches that worked just a few years ago are now basically useless," said Kevin Curran, senior member with IEEE.We also recommend reading security reporter Brian Krebs's take on this.

Comments

First lesson

[unixisc]

Make your website IPv6 only, so that DDOS attacks would have to be totally re-engineered to target them, and that too will be a tall order.

Re:First lesson

[BenFranske]

You can't just say that, please explain your reasoning.

Re:First lesson

[unixisc]

Why? DDOS attacks would then have to target the entire /64 subnet, which would be no mean feat

Re:First lesson

[darkain]

How exactly would being on a /64 prevent such an attack against a publicly facing entity? These attacks are not address space scanning attacks at all, they are known and publicly published IP addresses (in this case, DNS servers). Flood the public facing IP (the DNS server) would be exactly the same if IPv4 or IPv6. The only thing this would temporarily mitigate is the fact there are far fewer devices/users on the IPv6 network, so less of a botnet to control currently.

Re:First lesson

[Jamie Lokier]

Wrong. This type of attack targets known IPs of public servers, directly or indirectly.
A bigger subnet doesn't help.

Re:First lesson

[unixisc]

But on the IPv6 network, you have the potential to have thousands of DNS servers, or even multicast/anycast addresses for DNS servers. Not that many on IPv4, where you are short of addresses, and where you can't use private IP addresses for DNS servers.

Re:First lesson

[unixisc]

True, but on a /64 network, a server need not be restricted to one address. Like if you click on a link, it could redirect to another virtual host instead of a sub-directory, and here, the virtual host can use a different IP address instead of sharing it as is done in IPv4. There are several ways one could mitigate this issue

Re: First lesson

[Anonymous Coward]

The servers were just fine, the DNS was the problem.

Process:

0. user enters xyz.com
1. lookup Ip address
2. connect to ip
3. serve content

step 1 was the problem. all else was fine. you 'solution' addresses step 2 which however was not the problem here.

No. Step 1 was the problem. Stop whining alright.

Re: First lesson

[hackwrench]

Twitter has new content coming in in real-time, but once that content is created, it is as duplicatable as the data that has infrequent updates.

Re: First lesson

[dilvish_the_damned]

If your app can find the right server for the service, so can the attacking software.

Re:First lesson

[m.dillon]

I have two major beefs with IPV6. The first is that the end-point 2^48 switch address space wasn't well thought-through. Hey, wouldn't it be great if we didn't have to use NAT and give all of those IOT devices their own IPV6 address? Well... no actually, NAT does a pretty good job of obscuring the internal topology of the end-point network. Just having a statefull firewall and no NAT exposes the internal topology. Not such a good idea.

The second is that all the discovery protocols were left unencrypted and made complex enough to virtually guarantee a plethora of possible exploits. Some have been discovered and fixed, I guarantee there are many more in the wings. IPV4 security is a well known problem with well known solutions. IPV6 security is a different beast entirely.

Other problems including the excessively flexible protocol layering allowing for all sorts of encapsulation tricks (some of which have already been demonstrated), pasting on a 'mandatory' IPSEC without integration with a mandatory secure validation framework (making it worthless w/regards to generic applications being able to assert a packet-level secure connection), assumptions that the address space would be too big to scan (yah right... the hackers didn't get that memo my tcpdump tells me), not making use of MAC-layer features that would have improved local LAN security, if only a little. Also idiotically and arbitrarily blocking off a switch subspace, eating 48 bits for no good reason and trying to disallow routing within that space (which will soon have to be changed considering that number of people who want to have stateful *routers* to break up their sub-48-bit traffic and who have no desire whatsoever to treat those 48 bits as one big switched sub-space).

The list goes on. But now we are saddled with this pile, so we have to deal with it.

-Matt

Re:First lesson

[MightyMartian]

NAT may do a good job obscuring internal topology, but it does saw at considerable cost; breaking the end to end concept of the original ARPANET structure, requiring more resources, and creating far greater complexity for routers. Yes, a flat address space that sits in the public address space might, on the surface, expose more devices, but this is where firewalls come into play. I can still have a rather complex topology, but now I have to worry less about routing and connection tables, and can use less resource expensive techniques like tagging.

It was never IPv6's intention to be more secure, and you're right that many existing issues will remain with IPv6, and there will likely be new ones, but one thing is certain, if the solution is NAT, then that solution is worse than the disease it purports to cure. And it isn't as if NAT can't be vulnerable in its own way, and the only way to make it less vulnerable is, you guessed it, firewalls, authentication, and other security measures which are also needed in an IPv6 world.

Re:First lesson

[JustAnotherOldGuy]

IPV6 can make some things worse, especially spam.

For example, it makes banning or classifying an IP address as a spam source nearly impossible. There's so much address space that spammers will be able to use an IP to send 1000 emails and then discard it, never to be used again. The incredibly huge address space makes this quite practical. Banning by IP address will become meaningless because there are so many useable (and therefore discardable) IPs.

How much address space is there? Well....

Let's assume every single one of the 100 billion stars in the galaxy is inhabited, and each star has a population of 10 trillion humans in orbit around it, and each human has 1 billion devices that need IP addresses.

In that case, only 1/340,282nd of the possible 128-bit IPv6 addresses would need to be assigned.

Put another way, IPv6 would (will) provide roughly 5,000 assignable IP addresses for every square micron of the Earth's surface.

Hell, they could use one IP address per spam and never run out of fresh IPs in our lifetime.

Re:First lesson

[sabri]

go back to the whiteboard

APK was right all along! C:\WINDOWS\HOSTS is the solution ;)

*goes to hide under a rock*

Re:First lesson

[BenFranske]

+1 There is so much undeserved hate for IPv6 because people haven't taken the time to understand it.

NAT is not a security solution. If you would put a NAT device between your network and the Internet you can put a firewall between your network and the Internet. Yes, someone could potentially learn a small amount about your internal topology, well if you call being able to identify possible subnets withing your network learning about the topology, but the little they can learn is of dubious use. You still have no idea how most of those subnets are connected to each other (if you disable ICMP at your firewall or otherwise block tracerouting of your network from the Internet you can even prevent more) and even if you did please explain what substantial advantage an attacker has knowing how subnets are connected? If they're going that far it's an APT attack against your organization directly and you're probably done for because they will likely just trick someone inside the organization into installing malware on the network allowing them inside access and you'd have the same problems on IPv4.

Most of the rest of the list sounds like whining about more things you would have liked to have seen done, not things that are actually worse in IPv6 compared with IPv4.

Re:First lesson

[BenFranske]

Except that in reality the way it works is that each customer of an ISP is assigned a network block of IPs. If you find that customer is spamming you could block the entire network block. This is effectively the same thing as blocking the single IPv4 address assigned to a customer. The spammer would either need a new block of addresses from the ISP or a new ISP, effectively the same situation you have now with IPv4.

Re:First lesson

[JustAnotherOldGuy]

Except that in reality the way it works...

Except that in reality some ISPs are owned by the Russian Business Network (RBN), and they'll be given 100 million IPs to play with, and then another 100 million, and so on. The RBN owns lots of ISPs that are known for their friendliness towards "bulletproof" hosting companies and for working hand-in-hand with spammers.

-

The spammer would either need a new block of addresses from the ISP or a new ISP, effectively the same situation you have now with IPv4.

No, it's not the same situation because the address space that will be available to these criminal ISPs will be magnitudes of order larger than with IPV4. An ISP now may have a hundred thousand IPs to allocate (if that many), but now they'll have tens or hundreds of millions.

Seriously, this is going to be a problem, and more than a few security professionals have been discussing this problem for a while now.

Re: First lesson

[buchanmilne]

"But on the IPv6 network, you have the potential to have thousands of DNS servers, or even multicast/anycast addresses for DNS servers."

Most large DNS deployments already use IP Anycast on IPv4.

For example, Google's public recursive DNS (8.8.4.4, 8.8.8.8) uses IP Anycast. Most DNS root servers use IP Anycast.

There are two main benefits to IP Anycast, but the most relevant is allowing the distribution of an IP address over multiple geographic location, which allows lower latency, but also limits the number of attackers who can attack a specific deployment.

Re:First lesson

[unixisc]

But most ISPs do provide their own DNS servers - they don't just rely on 8.8.8.8. If you had all the DNS servers in the world, all part of a multicast to something like ffff::d5 or something, then it would be impossible to take down targeted websites via this route.

Re:First lesson

[unixisc]

The IP cycling that you suggest is something that I can see happening in IPv6. Have a mechanism in DHCPv6 whereby the PAM can cycle certain addresses to the domain name every x minutes/hours. That cannot be done in IPv4 due to address exhaustion - one can't use private addresses here under NAT

Re:First lesson

[unixisc]

As I've pointed out in past IPv6 threads, using the lower 64 bits is overkill. No subnet is ever gonna come close to even 32 bits, but in the meantime, you're limiting the hierarchical routing upstream in the global prefix that could have used some 16-32 bits more. In other words, had we had a split of 96:32 instead of 64:64, that would have made more sense. The subnet addresses could have used 16 bits, so that you'd have had a global prefix of 80, subnet addresses 16 and lowest 32 bits the interface ID. And one can still do auto-configuration under that system

Re:First lesson

[unixisc]

I have two major beefs with IPV6. The first is that the end-point 2^48 switch address space wasn't well thought-through. Hey, wouldn't it be great if we didn't have to use NAT and give all of those IOT devices their own IPV6 address? Well... no actually, NAT does a pretty good job of obscuring the internal topology of the end-point network. Just having a statefull firewall and no NAT exposes the internal topology. Not such a good idea.

The second is that all the discovery protocols were left unencrypted and made complex enough to virtually guarantee a plethora of possible exploits. Some have been discovered and fixed, I guarantee there are many more in the wings. IPV4 security is a well known problem with well known solutions. IPV6 security is a different beast entirely.

Other problems including the excessively flexible protocol layering allowing for all sorts of encapsulation tricks (some of which have already been demonstrated), pasting on a 'mandatory' IPSEC without integration with a mandatory secure validation framework (making it worthless w/regards to generic applications being able to assert a packet-level secure connection), assumptions that the address space would be too big to scan (yah right... the hackers didn't get that memo my tcpdump tells me), not making use of MAC-layer features that would have improved local LAN security, if only a little. Also idiotically and arbitrarily blocking off a switch subspace, eating 48 bits for no good reason and trying to disallow routing within that space (which will soon have to be changed considering that number of people who want to have stateful *routers* to break up their sub-48-bit traffic and who have no desire whatsoever to treat those 48 bits as one big switched sub-space).

The list goes on. But now we are saddled with this pile, so we have to deal with it.

-Matt

The first point about NAT - while that used to be a shortcoming in terms of topology masking and load balancing, the IETF did explicitly define Network Prefix Translation, which is a 1:1 NAT mechanism that would do what you want, but avoid the pitfalls associated w/ the 1:many mapping in IPv4 NAT. Also, IPv4 NAT consumes several port addresses, and also often several NAT layers, reducing the networking to layer 2. As for IPSEC, I didn't exactly get your point on why that is a bad thing. As for the subnet port scanning, I happen to think that having a DHCP-PAM setup where addreses can be set to change regularly would be more effective at preventing a breaking, rather than relying on the brute force of trying to prevent a port scan of /64. But this is something that can be done in IPv6, where you have no address shortage within your subnet, as opposed to IPv4, where chances are you wouldn't have > 256 addresses to play w/, unless you happen to be IBM or HP or one of the early recipients of public Class A blocks.

Also, on the 48 bit space - I somewhat agree w/ you that assigning 64-bits to the interface ID was way overkill. But if you are assigned a /48, you can have 65536 subnets of /64 within that - that's what the convention allows. Breaking it up more is what may create problems, which again, I disagree w/. The main thing about the subnet address assignment is that the moment you want to lend structure to it, the number of effective subnets you can create goes down. Which is why I wish they had made the entire top half the global prefix, and then split the bottom half b/w the subnet address and interface ID. Something like a 16:48 would have been ideal, or else, even a 32:32 if one needed plenty of structure from the subnet addressing plan.

Re:First lesson

[unixisc]

Assuming that all the addresses coming out of, say, Russia, are suspect, one can look up RIPE's address assignments to see which addresses they have been assigned, and block them. In fact, this is something that one can do pretty easily if one needs to block out a country. Like you think all the addresses from Syria are owned by ISIS? Check out RIPE, see which blocks have been assigned to Syria, and instruct your firewall to drop all their packets. This can be done as high up as ISP level

Re:First lesson

[Dagger2]

They'll have more address space available, but it'll still be in contiguous blocks. If an ISP as a whole is being a problem then all you have to do is block their v6 allocations, which is no harder than blocking their v4 allocations. (Or possibly easier since the ISP is likely to have a single v6 allocation vs dozens of v4 ones.)

Re:First lesson

[JustAnotherOldGuy]

If an ISP as a whole is being a problem then all you have to do is block their v6 allocations, which is no harder than blocking their v4 allocations.

You're talking about blocking 10 million or maybe 100 million IP addresses, and in that range are going to be some (or even many) legitimate users. Also, there's no guarantee that the address space will be contiguous, it may be broken up into various blocks. If they have their way then it will almost certainly be spread across many, many different blocks of IP addresses.

And finally, in addition to being spread out over many ISPs and many blocks, they'll almost certainly be using IP-shifting, fast-flux, or other masking techniques that will make this a whack-a-mole problem that anti-spam services will never be able to keep up with.

These people aren't amateurs, they know what they're doing, they have lots and lots of money and talented people at their disposal, and they're thinking ahead. And that's just the RBN.

To think that this is going to be solved by blocking a few ranges of IPs is really kind of naive.

What's the Solution?

[BenFranske]

I've heard a lot of people today saying there's a problem. Several of the commenters (on Brian Krebs' blog for example, on the NANOG list for another, and probably soon here on ./) say we should do something to fix this so it doesn't happen again. What I haven't heard is a real proposal about what to do about stopping DDoS attacks.

Re:What's the Solution?

[Anonymous Coward]

I've seen it a million times, in no small part because I've posted it myself:

ISPs need to start egress filtering to block spoofed packets coming from end users with forged source addresses. If a packet comes from joe blow's cable modem with a source IP from some other country, it should just be dropped.

Re: What's the Solution?

[Anonymous Coward]

A good start would be to criminalize the manufacture and sale of iot devices with little or no security.

Re:What's the Solution?

[BenFranske]

Yes, this is effective against some subset of attacks. There was a good reminder/discussion of this on the NANOG list this morning. The problem is 1) probably pretty much every ISP which can be convinced to do this is already doing it at this point, the others are probably a lost cause and 2) this only prevents attacks where the address actually is spoofed. If a large number of compromised devices are running malware they can just make an overwhelming number of legitimate service requests en masse...

Re: What's the Solution?

[BenFranske]

1) Yes, poorly designed IoT devices make the problem worse but it's existed long before IoT came along. 2) What qualifies as an IoT device, every Arduino with an Ethernet/WiFi port? The code isn't on them until you program them... 3) If mass regulation of all network connected products is the only way we have a problem because you're never going to get global agreement on that and it's going to be nearly impossible to enforce.

Re:What's the Solution?

[rudy_wayne]

( 2) this only prevents attacks where the address actually is spoofed. If a large number of compromised devices are running malware they can just make an overwhelming number of legitimate service requests en masse...

Why would an ISP allow me to make "an overwhelming number of legitimate service requests"? Oh, that's right, you answered that question in point #1 -- most ISPs don't give a shit.

Re:What's the Solution?

[guruevi]

A) Avoid a single point of failure (the cause of downtime across all these providers)
B) avoid using a single point of failure
C) stop using public DNS (or DNS at all) for self-configuration and discovery of your hosted servers
D) stop using a single provider for all your stuff

Re:What's the Solution?

[guruevi]

Not how the Internet works. Yes that's true on the edges but once you enter into the public Internet, packets could be routed from anywhere to anywhere. The only solution here is to shut down ISPs that are participants but you're talking about getting participation from people that often are themselves involved in the criminal enterprise (that's true for US, Europese, Chinese etc providers) and are profiting from these attacks through overage fees etc.

You wouldn't imagine but even providers like Verizon won't shut off mobile connections because they are often charging their customers per GB consumer. A lot of sleazy hosting provider (the cheap $5/mo.VPS) simply delays intentionally or unintentionally because they don't have the staff to keep up and they are often paid for by the criminals.

Re:What's the Solution?

[Penguinisto]

...and #2 is going to become a *lot* more common, thanks to growth in IoT. :/

Wish they'd have paid more attention to crap like this back in the late 90's when the whole idea first surfaced on a serious note (e.g. JINI).

Re:What's the Solution?

[Archangel Michael]

It is worse than that.The problem with DDOS is that the real victim probably doesn't know about it.

The proper way to thwart these kinds of attacks is to have a method of detecting them and then cutting off people who are making an inordinate amount of those kinds of packets aimed at that address. The solution to a coordinated attack is a coordinated response.

Re:What's the Solution?

[BenFranske]

I would maintain that's not possible. Attackers will just write software that mirrors normal user traffic accessing a site. It's simply the fact that millions of devices will be accessing the site at the same time that takes the site/service down. Just like ye olden days when nearly every site mentioned in a ./ summary went down. The fundamental problem is that a truly distributed denial of service attack is just a coordinated accessing of a site from a large number of hosts. The only difference between that and just a lot of people visiting your site is that one is coordinated. Good luck detecting the coordination.

Re:What's the Solution?

[Anonymous Coward]

It would make life very difficult for the attackers.

Attackers would have to reduce the rate at which they access the site from any single bot in their botnet, otherwise the bot sticks out, can be identified and cut of the internet. That means they need more bots.

At the same time, if ISP's finally start to follow up on abuse notices, the number of bots would actually be reduced.

Together these would greatly reduce any botnets power.

At the same time we only need someone to maintain a list of ISP's who refuse to clean up their act. Everyone who does care about this can then blackhole those IP addresses. Or, in the case of a website, refuse access with a message explaining why.
If enough websites and internet services do this, the bad ISP's will get in serious trouble. Pissed of customers are bad for business. ow they have an incentive to clean up their act. It worked for spam too, many ISP now simply block the smtp port by default.

Re: What's the Solution?

[hackwrench]

The key to understanding wat is an IoT device is in the word thing. Devices like network controllable light bulbs aren't multipurpose devices like a regular computer or an Arduino.

Re:What's the Solution?

[turbidostato]

"The proper way to thwart these kinds of attacks is to have a method of detecting them and then cutting off people who are making an inordinate amount of those kinds of packets"

Unless there're no inordinate amount of those kinds of packets but an inordinate amount of clients requesting usual amounts of packets each. That's the first D on DDoS, after all. How can you distinguish a malicious DDoS from the Slashdot-effect of yold?

On the other hand, this was not a DDoS attack targeted against the servers themselves but against the DNS that allow clients to find them.

"The solution to a coordinated attack is a coordinated response."

It really depends on the nature of the attack. History shows that against a coordinate attack just entrenching may very well be the proper counter.

Re:What's the Solution?

[Zero__Kelvin]

How many different ways can you say the same thing?

Re:What's the Solution?

[Dutch Gun]

How many different ways can you say the same thing?

A) Several ways
B) A lot of ways
C) Nearly infinite ways
D) About four, it seems

Re: What's the Solution?

[BenFranske]

An Arduino is just an AVR microcontroller, the same chip found in many electroinc/IoT devices. Point being when does it become an IoT device? If I sell it? How about if I just sell it to a few friends? Maybe I make and sell a small quantity on etsy? etc. It's hard to draw a line about when it's an IoT device and when it's just me playing around with electronics.

Re:What's the Solution?

[flabman]

These attacks cause service outages because legitimate DNS lookups can't be handled by the servers that are under attack (which I'm assuming here to be the authoritative name servers for the domains that are experiencing service outages). Most users don't ever query the authoritative servers directly; the legitimate queries come from their ISPs' resolvers, and those resolvers only query the authoritative servers if they don't already have the answer in their local cache. And that only happens (in respect of popular sites) when the cache entry's time-to-live has come and gone.

So perhaps one way of at least partially mitigating these attacks is for resolvers to hang onto cached records past their TTL and to continue serving them when the authoritative name servers are unavailable. Those resolvers will then of course need a robust alternative cache ejection policy (e.g. based on the frequency with which an expired record continues to be used, how overdue it is, and overall resource usage).

I do realise that Dyn is also known for their dynamic DNS service, and that the above mitigation isn't effective for ephemeral records which intentionally have a short TTL. That can't be helped.

Look at the Bright Side

[tmjva]

At least the hackers didn't bother shutting down Slashd...

Re:Look at the Bright Side

[MachineShedFred]

Well, we all know that Slashdot uses the mighty APK HOSTS engine to protect it!

Just hang around here long enough, you'll see everything you've ever wanted to know about it.

Re:Look at the Bright Side

[un1nsp1red]

Well, we all know that Slashdot uses the mighty APK HOSTS engine to protect it!

Just hang around here long enough, you'll see everything you've ever wanted to know about it.

Actually, it's been weeks or months since I've seen APK. Can we get a wellness check on that fella?

Re:Look at the Bright Side

[unixisc]

He responded above to my curiosity about his Russian heritage, so yeah, he's here

Flood defenses?

[m.dillon]

There is no flood defense possible for most businesses at the tail-end of the pipe. When an attacker pushes a terrabit/s at you and at all the routers in the path leading to you as well as other leafs that terminate at those routers, from 3 million different IP addresses from compromised IOT devices, your internet pipes are dead, no matter how much redundancy you have.

Only the biggest companies out there can handle these kinds of attacks. The backbone providers have some defenses, but it isn't as simple as just blocking a few IPs.

-Matt

Re:Flood defenses?

[Dadoo]

but it isn't as simple as just blocking a few IPs.

And this is why people need to be fined, if a device on their home network is found to be part of a botnet. Individuals need to be responsible for their networks, because the authorities are virtually powerless against botnets, Unless it costs them money, people just won't care.

Re:Flood defenses?

[0123456]

And then what?

They buy a device that's horribly insecure, the manufacturer sends out one security update, then abandons it, and it becomes part of a botnet. And you fine the person who bought it?

Actually, you're right. That's a great idea, because it will kill the whole 'Internet of Things' idiocy overnight. No-one will risk attaching anything to their network if they can't verify it's secure.

Re:Flood defenses?

[Dadoo]

That's a great idea, because it will kill the whole 'Internet of Things' idiocy overnight. No-one will risk attaching anything to their network if they can't verify it's secure.

Well, that's one potential side-effect - and not necessarily a bad one, in my opinion. Either they learn how to manage their devices, or don't connect them to the Internet.

Tackling Mirai

[subk]

Now that the source code for Mirai is out there being used, is there something that can be done to tackle the spread? Call me crazy, but perhaps a modified version could go out and actually change the passwords on these insecure IoT devices to random strings? Sure, the owner would lose access to the device.. But it would alert them that something was wrong, and stop the spread of Mirai.

Re:Tackling Mirai

[ArylAkamov]

1. Modify it to use all infected devices as a giant neural network
2. Resurrect Tay A.I.
3. ???
4. Bow down to our Nazi A.I. overlord

Re:Tackling Mirai

[citizenr]

reuse infecting part, replace bot part with firmware updater flashing data straight from /dev/urandom
brisk every single alliexpress special $20 web camera out there, repeat every 3-5 months when new hardware comes out until Public learns NOT TO PUT GARBAGE on the public internet

DNS Replication Service Suggestions?

[codebot]

Anyone have recommendations for a good DNS replication service?
Would prefer to be able to replicate rather than maintain two sets of data.
A search turned up www.buddyns.com, but I've not yet dug into their details yet.

Re:DNS Replication Service Suggestions?

[Qzukk]

How about http://dyn.com/secondary-dns/ ?

Re:DNS Replication Service Suggestions?

[OverlordQ]

Why do you need a replication service? If your stuff is automated, just point it to two providers.

Re:DNS Replication Service Suggestions?

[codebot]

Dyn is who I want to replicate.

Re:DNS Replication Service Suggestions?

[guruevi]

I think EasyDNS has a product but it's as simple as maintaining two sets of DNS records and pointing your domain to two different providers (e.g. powerDns and easydns).

This "attack" could've been easily prevented if they had a single SysAdmin with 15-20y experience in Internet hosting. Having multiple DNS providers used to be standard practice for any medium to large organization.

Imagine dyndns CEO or disgruntled employees simply pulling the plug out. Same result and a reason to avoid SPOF even if you're "in the cloud"

dare I say it? APK APK APK!

[Anonymous Coward]

As a backup everybody should maintain a HOSTS file with every internet address ever in it.
Can't DDOS that.

Lameness filter encountered. Post aborted! Filter error: Please use fewer 'junk' characters.

Proof the system is rigged...

This is a test. This is ONLY a test.

[cfalcon]

If this had been an actual attack, all internet services would be rendered inoperative for long enough for whomever the fuck is doing this to have accomplished whatever the fuck awfulness they desire.

Re:dare I say it? APK APK APK!

[unixisc]

Are the Russkies behind these DDOS attacks? If yes, nothing like an APK solution to fix it. A Russian solution to a Russian problem!!!

Re:dare I say it? APK APK APK!

[snookiex]

Since my ISP stopped allowing me to access the admin console of my modem and started exposing a remote management interface to the internet, I don't trust anymore the DNS information provided via DHCP. Probably using a VPN service would be more practical, but for now, I use the hosts file for the sites that require authentication.

Re:dare I say it? APK APK APK!

[barbariccow]

HOSTS file

sshhh... If you say H***** file 3 times in a mirror you'll summon APK... DO NOT ATTEMPT!

Is it really a war?

[beheaderaswp]

I've been looking at the mainstream media outlets and they are reporting on this attack as if we were just invaded by Russia.

This was an attack against DNS... at worst this type of attack stops people from "doing something". That "something" could be playing Pokemon... or banking... or working. But it doesn't "take down" the internet.

The internet is just fine. To take down the "whole internet" you'd have to attack routers. And the numbers of routers exceed the ability of anyone to saturate them. So why does the media get all hyped up when Twitter goes down?

It irks me so badly that the media and the general public get so completely flustered when some third world country, or a group of kids, decide to play games with the system. And that is all it is.

Certainly we should defend against disruptions like this. How they are done should be researched. Perhaps in the future the system can be hardened so it's incredibly difficult to attack it.

But it's a pretty minor league attack against the "internet". Twitter is down? The NYT?

I just turned 50 last year. Still up to date on tech. Still as sharp as I was at 25 when I lugged a Compaq suitcase around. This seems like such a small issue to me. When the real issue should be router security, the idiotic idea of tying SSL certs to domain names, or the sad security of home routers.

Re:Is it really a war?

[DerekLyons]

This seems like such a small issue to me.

That's because you just handwave away the issue, mostly by pedantically nitpicking the terminology.

Re:Is it really a war?

[0123456]

So why does the media get all hyped up when Twitter goes down?

The media thinks Twitter is The Internet.

The real point of these attacks is that DNS, like any other centralized service on the Internet, is broken by design.

Re:Is it really a war?

[beheaderaswp]

Yes I do handwave the issue- because it's a small one.

If you want to talk about big ones, I can go there as well. The US could deploy an alternative DNS system in days. Either with the current tech or something truly distributed.

That's the real issue. The press thinks this current attack is important. And what does the public do with this information? Are they going to revise the system?

If I had my druthers the whole DNS system would have been trashed around 2005 and replaced with a blockchain that would have a node density so high it could not be attacked effectively..

So if you think that's nitpicking terminology I fear you are far less competent on these issues than your low number on this site would infer you to be.

Re:Is it really a war?

[c]

But it's a pretty minor league attack against the "internet". Twitter is down? The NYT?

I was just reading a Facebook comment from a friend about a hospital basically shutting down... presumably they had a dependency on something "in the cloud".

Now, I'll certainly grant that said hospital fucked up beyond belief by having that dependency, and I'd hope that heads will roll over it, but the impact seems to go beyond mere entertainment.

Re:Is it really a war?

[Dynedain]

"The Internet" hasn't meant the physical network for at least 2 decades. Since at least the early '90s and the "internet superhighway", average people have used "The Internet" to refer to the collective set of interactive services and activities made possible by the network, rather than the underlying network hardware itself.

What good is the physical link if nothing intended to run on it is actually functioning?

Re:Is it really a war?

[turbidostato]

"I was just reading a Facebook comment from a friend about a hospital basically shutting down... presumably they had a dependency on something "in the cloud".

Now, I'll certainly grant that said hospital fucked up beyond belief by having that dependency, and I'd hope that heads will roll over it, but the impact seems to go beyond mere entertainment."

Wont' happen. Executives *love* "the cloud" because, among other things, it can very effectively deflect blame. It was not me, it's been DynDNS, a reputable actor in the industry, who would have expected it! (by "who" being other executives, of course, not somebody with actual technical acumen). So the most that will happen is that they'll go from this sole provider for that service to another one with even higher "corporate image".

Re:Is it really a war?

[mars-nl]

It doesn't have to be centralized. Hierarchical, yes, centralized no. Putting all your eggs in one basket (Dyn) is just not a good idea. People outsource stuff and then stop thinking. People assume that if they outsource to a company, nothing can go wrong. But big companies are bigger targets and when they fall over, the mess is much bigger. So yes, decentralize.

Re:Is it really a war?

[thegarbz]

But it doesn't "take down" the internet. The internet is just fine.

Sorry but this very serious and you ignore the realities of the modern internet which is as much dependent on DNS as it is those routers. DNS isn't just a name lookup service anymore. You can't fall back to something else when DNS is down. Many devices and programs come with hardcoded domain entries. Much of the internet is now wholly dependent on DNS to correctly localise services or even know what content to serve you. www.siteofinterest.com may resolve to a specific IP address, but good luck typing in that IP address and getting to that site which may be sitting in a datacentre hosting any number of virtual servers.

The internet was not "just fine" yesterday.

Re:Is it really a war?

[c]

So the most that will happen is that they'll go from this sole provider for that service to another one with even higher "corporate image".

I believe it's a Canadian hospital, so its executives might have a different sort of accountability. I hope.

Re:Is it really a war?

[turbidostato]

"I believe it's a Canadian hospital, so its executives might have a different sort of accountability. I hope."

They most probably don't. And even if they do, that won't be the case for long. The nice thing about globalization is that it is a race to the bottom. In this case it translates to -how we Canadians can be competitive if our executives have higher accountability than their USA counterparts?

Yet more proof and confirmation that...

[X86BSD]

CLOUD anything and outsourcing your infrastructure because you are lazy and/or cheap is a BAD IDEA. Consolidating services you no longer control to a third party means you've lost the ability to survive these attacks.

Re:Yet more proof and confirmation that...

[beheaderaswp]

I so totally agree.... I'd mod you up but I posted and can't.

It was WikiLeaks (supporters)?

[CustomSolvers2]

At least, this is what they said in their Twitter account.

Re:It was WikiLeaks (supporters)?

[CustomSolvers2]

Apparently, I wrote this post too soon, because it didn't become news until some hours later.

November 1987, RFC 1034. Secondary DNS servers

[raymorris]

For this specific attack, set up a secondary name server, using a secondary provider.

In November 1987, RFC 1034 was published. It describes how secondary DNS servers automatically sync from the primary. For about twelve years, people took that seriously. The used ar least two name servers that were unlikely to be affected by the same problem - separated geographically far apart and using two (or more) different network providers. Nowadays it's likely their two name servers are sitting right on top of each other in the same rack.

If both your DNS servers are with the same provider, wherher that be Amazon, DynDNS, or any other single provider, they are subject to fail due to the same cause, at the same time.

Btw ona different, but related topic - there's also an RFC for exactly how to build CDNs (reverse proxies) that actually work right. We've known how to do that correctly for decades, so everybody can read the damn RFC and stop inventing new ways to completely screw it up. First hint - the protocol for reverse proxies has been around far longer than the buzzword "CDN" that's now used to sell them.

Real Business Implications of Internet 3

[WillAffleckUW]

Look, when we built the Internet (back in the ARPA days), it was restricted to trusted players at military and research universities.

Then we let in the unwashed masses.

Then some morons decided to give Internet capability to every single device in the Internet of Things.

First principles, people:

Build one Internet based on IPv6sec for the trusted peers. The backbone.

Build a second Internet for the identified non-object computers based on IPv6sec. The unwashed masses. If parts misbehave, turn off their feeds until they fix them. Drought solves lots of problems.

Build a third Internet for the Internet of Things based on IPv6 and IPv4. Restrict the ports and traffic to essentials. So you can't play Disney in your car, too bad.

Re:Real Business Implications of Internet 3

[bn-7bc]

Well yes, but(without putting words into gps mouth) I think she/he reffered to people who could not separate udp from tcp if their life depended on it. You now the kids nd if people that says the internet is down when they cant get to ( insert one populare service here) without checking if any other service is still working

Solution?

[mrPalomar]

Is it time for blockchain DNS?

Re:Solution?

[guruevi]

No, just DNS the way it was intended. DNS and all early Internet services were designed to withstand nuclear war and attacks by state-sized actors, actually specifically designed to withstand an attack from Russia.

The problem is the cloud has aggregated all that diversity of everyone running their own services into a handful of really big corporations. Today's just a reminder that any one of those corporations has a significant amount of control if it were a truly bad actor. Imagine Dyn intentionally pointing all the Twitter etc DNS records elsewhere, they did it for their "free" accounts a decade ago just to make them pay.

It seems no one at those big corporations remembers the true history of DynDNS, and how they screwed their customers over. I was surprised they were still in business at all.

Re:Solution?

[toonces33]

Just turning off or filtering DNS UDP packets would be a start.

DNS over UDP works fine on an intranet. Just block it on the way out onto the rest of the net.

Re:Solution?

[BenFranske]

Actually, in cases like this it would make it worse. This is not the DoS of your youth with spoofed IP addresses. This is millions of bots making seemingly legitimate requests simultaneously. With UDP DNS requests are a single packet. With TCP you get a SYN, SYN ACK, and SYN before you even get to the part where you're making the query...that would dramatically multiply the number of packets for each query from each bot, or for that matter on a regular day from a legitimate user meaning the connections would just be that much closer to being flooded all the time.

Reddit, Spotify, and the New York Times were down?

[clonehappy]

There's only one answer: Activate Homeland Security!

You know, the department established to thwart terrorists who plan on mass murdering people in spectacular displays like knocking down skyscrapers? The one plenty of us told you was going to be used for every crime in the book in addition to terrorism?

Nah, that kind of scope creep would never happen I was told...

OpenDNS SmartCache

[Jayfar]

I was reading elsewhere that users utilizing OpenDNS' SmartCache feature were unaffected. Basically, in the event that a domain's authoritative servers all become unavailable, smartcache uses the last known good resource records, regardless of whether their TTL has expired. Are any of the other DNS providers and ISPs utilizing anything similar?

Re:Russians!

[Anonymous Coward]

Naa aar, I heard it was Hillary trying to delete those last few emails from her server.

APK is pure

[unixisc]

APK is a pure unix/linux guy ;-)

Re:No Russian ancestry here... apk

[unixisc]

You're talking about the grand duchy of Poland-Lithuania? That included what's today Belarus and the western parts of Ukraine. Saying that Poland took 'Russia' seems to suggest that they took the entire country. At the time in question, I doubt that the Poles even had Moscow, much less the western Urals

Re:Russians!

[unixisc]

But why would the Russians then shut down Trump's main channel - Twitter?

Circular logic

[DerekLyons]

Yes I do handwave the issue- because it's a small one.

Thereby creating a circular chain of logic.

Dunning-Kruger Effect?

[Gazzonyx]

FWIW, the GP poster is Matt Dillon. He's a well known FreeBSD/Linux kernel hacker and the founder/maintainer of DragonFly BSD and his list of Nerd Cred is legit and long. I'm sure he's forgotten more about network protocols than I ever knew in spite of my kernel patches and Samba contributions. I'd wager he's painfully aware of the ins-and-outs of NAT and IPv6 at a low level.

Don't get me wrong; that doesn't mean he might not be wrong in his evaluation of the protocol. He'd just be wrong on a much more detailed level than I could comment on with any comfort. :)

Luddites

[Hognoxious]

They should get with the times and move from the olde worlde internet to the all new shiny shiny cloud!

Re:Anyone else did the same OR better?

[unixisc]

You are right about the Poles under Sobieski repelling the Muslims at the gates of Vienna. But as far as the question of whether any nation conquered Russia in a major way, nobody did it more comprehensively than the Mongols - both under Genghiz Khan and under the Golden Horde.