Wouldn't you try a test run on one user account before letting it fly for all users?
You are absolutely right, that's how it should have been done. And you can take it in more steps. You don't have to go directly from one to all. Decide how many steps are appropriate and then increase the number of users exponentially between each step.
This sort of approach is standard practice at Google. I don't think Google made the decision to rush this through. I imagine Google was warning about the possibility of something going wrong, but Sky and Yahoo somehow reached an agreement, that Google was simply trying to hold onto the users for as long as possible. So they decided to ignore Google's "warning" and go ahead with a quick transfer of all the users. This is just my imagination of how it might have happened, I don't know if this is exactly how it happened.
A transition period where users are distributed across the two systems requires a special layer of frontends, which know which users are currently on which system. This adds complexity, and complexity increases the risk of bugs. For mails which are arriving via SMTP it is fairly easy to have frontends, which simply delivers copies to both Google and Yahoo. But for anything else it is hard to keep the systems in sync. Doing the entire migration in one go may have been choosen for simplicity. It is possible Sky wanted this approach because it was cheaper.
An email exists in more than one place. Deleting it from one system doesn't imply it is being deleted from all other systems. From the article it is not entirely clear which systems are in play, where it was deleted, and where it was not. From the description I get to think of two likely scenarios.
Users downloaded the messages from the server to their own computer using the POP protocol. On the Google side it was configured such that a delete requested over the POP protocol was interpreted as archiving the message. The user then deleted the message from their own computer without thinking about the copy still in the archived mails "folder" on Gmail. Somebody messed up as the messages were transferred, and they all became available over POP again, and thus were downloaded again.
The problem was on the sender side with messages in the Sent folder being misinterpreted as Outbox from where they were sent and then placed in a Sent folder. As old messages were being sent again receivers who had already deleted them are complaining as they show up again. They really had been deleted on the receiving end, but still existed on the sending end.
Duplicate MACs have no bearing on port-security (the switch honestly doesnt care, as long as your MAC doesnt mysteriously change).
The documentation from Juniper disagrees with you (first hit when searching for sticky MAC). I do consider documentation from a major vendor of switching hardware to be more trustworthy than comments on slashdot from people I know nothing about.
If you had such a configuration, the routing table on your private router / computer should be set up to send packets with the correct external address out via each interface.
You can't change your IP in the middle of a TCP connection (though that would be a really nice feature). And some websites will break if you switch IP address in the middle of a session. YouTube streaming has been seen doing that.
If you fail to send traffic over an interface without using the correct interface address, the ISP should rightfully drop that packet.
If they don't, they will be doing a service to those customers, who do have redundant links. There is nothing in IP mandating packets take the same path in both directions.
And those packets are only allowed to be sent directly to an IPv4/IPv6 relay router. Meaning that if your ISP does not implement such a relay, they are still right to disallow such packets from exiting their network, as there would be no way to route return traffic.
I can see you don't know as much about that protocol as I do. But I have written an implementation of the protocol, so I should know how it works.
First of all, the packets I mentioned are not being sent to the relay, they are being sent from the relay. And anybody is allowed to setup such a relay. On a simple home network, you cannot advertise your relay over BGP (and you probably wouldn't want to), but it can service your own LAN without being advertised. So you can have a legitimate relay on your home network. The packets it sends must of course be sent to 6to4 border router, but that can be many hops away, it certainly doesn't have to be within your ISP's network. And the destination could be on any globally routeable IPv4 address (except from 192.88.99.0/24).
I'm not sure why one would use DSR for such a set up
DSR will give you much better performance than an equivalent setup without DSR.
A responsible ISP would also require that you provide written proof for each range that you wish to transmit on behalf of.
It is reasonable to only enable it if customers ask for it. But they can't make the process more complicated than signing up in the first place without pushing some of those customers to other providers.
Hosting providers need someone to blame for all traffic, if they allow your traffic without a written arrangement, their ass is on the line should it be traced back to them.
Doesn't look like it.
There are other methods of determining this as well, however, if you are dealing with an ISP, complain, test via VPN, and if they don't remove the limit, drop them.
Traffic would take a totally different path across the network. A difference in responsiveness across the two paths would be expected, and it does not prove that the ISP is treating traffic differently.
You're presenting a lot of edge cases
The point isn't whether those are corner cases, the point is how many legitimate use cases I could present right off the top of my head. I think there are plenty more use cases, which I haven't thought about. And maybe some of those use cases will never be fully thought out, simply due to the person who would have come up with a clever idea dropping it because of being restricted by his ISP. Of course the cases I described are not typical, they couldn't be due to many ISPs dropping that sort of traffic. If that traffic hadn't been dropped, such use cases might have been typical.
This is not to say ISPs are doing something wrong by dropping packets they think have a spoofed source address. But neither should we say ISPs are doing something wrong by allowing it. For me it takes only one legitimate use case to defend those ISPs who do permit it. The root of the problem is not spoofing, but amplification. (I have experienced a case that would make a factor 100 amplification look harmless, unfortunately I am not allowed to go into details about that episode.)
Every port being on a different segment is utterly unnecessary
Completely eliminating problems with MAC spoofing is worth something. It would mean even if a vendor by mistake assigned same MAC to two devices, you could plug them both into the same router, and it would just work.
and would introduce unneeded latency
My calculations says the opposite. But it is really a minor difference, because it is only for the initial neighbor discovery, there will be a difference. Once receiving MAC addresses are known, the actual payload traffic takes the exact same physical path, hence no change in latency. For the neighbor discovery it is more likely for each node having an entry for the router in cache already and the router having entries for the devices, than it is for the devices having entries for each other. So the probability that you need to wait for a neighbor discovery is smaller. Moreover the latency of that discovery will be smaller, as it will need to go over only a single Ethernet link, rather than the two links needed in case of doing it between two devices each connected to a switch.
Which is a problem only if you are doing this using public IPv4 addresses. But the networks we are talking about here aren't using public IPv4 addresses in the first place.
and all for no benefit.
Unless you want the introduced security it offers to your LAN. Some attacks works only within the same segment. Putting every port on a separate segment means you only need to protect the router against such attacks and not every single device.
As I said, you cal ALREADY turn on port-security with dynamically learned MAC addresses on a mid-range Cisco Switch, which immediately thwarts MAC spoofing / ARP poisoning.
Those are not the only sorts of attacks that you can perform within a segment. And learning MAC addresses doesn't make sure your devices will actually be able to communicate in case the vendor screwed up and manufactured two with identical MAC addresses. Putting each port on a separate segment would solve that problem.
Routers perform a vastly different job than switches, and do so orders of magnitude slower. switching is performed essentially "wire-speed", with fractions of a millisecond incurred by switching delay
With the proper chips, routing can be done at wire-speed as well. It is not even much more complicated to do so. Of course what I mentioned won't happen until the devices are being manufactured with chips capable of doing it.
you STILL would have to do filtering, just at layer 3 instead of layer 2.
It is not something you absolutely have to do. IP is not that vulnerable to source address spoofing. Higher level protocols deals mostly fine with it.
The MAC filtering you proposed wasn't validating the source IP anyway. Even if you don't spoof the source MAC address on a packet, you can still spoof the source IP address and send it through a switch.
Filtering invalid source IPs would be easier in the setup I expect to see, because the router actually knows which source IPs are valid on which ports. In contrast the switch doesn't know which MACs are valid on which ports, and it doesn't even know what an IP address is.
It is still technically within the "switching" realm (as you are doing a lookup of physical port to IP address-- something that routers simply do not do)
Finding out which port a packet must leave on based on the destination IP address is the defining feature of a
if your main line goes down and you continue to use that IP on the redundant connection, your return traffic will still be routed to the downed line.
Of course it won't work if the line is completely down, or if it for some other reason is unable to deliver traffic to you. But if the line can deliver packets to you, and only the outgoing path is having a problem, then it can indeed help. He cannot give you any percentage of what kind of reliability improvement it could give, since I haven't had the opportunity to measure such a setup.
TCP will not work when the source ip is spoofed; the 3-way handshake will fail
I was not trying to imply you could do a TCP session working with a spoofed IP, quite the contrary. My point was the packets with spoofed IP address does not immediately break legitimate traffic on that IP. In fact it is quite difficult to break established TCP connections, if you are not on the path it flows through, even if you have access to spoof an IP address. That a server receiving incoming TCP connections can be fairly confident the client IP is authentic, is an added bonus.
On a switched network with non-managed switches, spoofing the MAC address of another computer may be enough to kick that other computer off the network. You need to use managed switches and configure ACLs on those to stop that sort of attack.
Switches will probably not be replaced by routers; they serve different purposes, and in fact you see the reverse trend-- layer 3 switches which are able to perform rudimentary routing.
If it performs routing, then it is a router. Hence if that's the way we are heading, I was right saying the switches will be replaced by routers.
All routers I have ever worked on explicitly forbid having more than one physical interface with the same subnet (home wifi routers are combo router / switches).
And I expect that to be taken one step further. There already exist chips, which can be either a switch or a router depending on how you configure them. And if you just configure such a chip to act as a router with every port being on a different segment, then you don't need to worry about MAC spoofing.
Yes, invent a new protocol. What a genius! Why did no one come up with this before?
Somebody did already, they just didn't do a very good job at it. EDNS made this sort of attack 8 times more effective than it was before EDNS was introduced.
If spoofed source IPs need to be allowed, they should only be allowed for customers with a legitimate reason to do so, and at the ISP's peril should the customer abuse such privilege.
That is a very sensible approach. It does however require a little bit more management for the ISP to do this, and they will likely charge extra for that. Customers who have a choice may decide to go with an ISP that permit it by default, because it is simpler than figuring out how to get it turned on, and probably cheaper too.
If that ISP decides to turn it off, and they actually have customers relying on it, they'll suddenly have to deal with support calls.
If it actually had a cost to the ISP, who was permitting spoofed source addresses, the situation might look different. But as it is, tracking down the source of packets with spoofed source is a bit tricky.
There is NO scenario where a wired internet connection should be spoofing IP addresses, because the ISP will NEVER be able to deliver return traffic to that person.
It could be delivered through a different path. For example you might have redundant internet connections through different providers. It could also be that one of the two paths is using VPN and the other is not.
There are also special addresses, which are valid as source address, even if they are not assigned to you specifically. For example, I could send packets through my ISP with source address 192.88.99.1, which would be perfectly legitimate and not perform any IP spoofing at all. But even though the packet is perfectly legitimate, and the reply would indeed be routed back to me, that probably won't work, because my provider is filtering and didn't know about 192.88.99.1, when configuring the filters.
If we look at hosting companies, there is another reason why a customer may need to send packets with apparently spoofed source addresses. It may be that the customer is simply doing load balancing across multiple data centers, and are using a DSR setup.
At some point I watched a talk mentioning another legitimate use case for apparently spoofed source address. It was for investigating what sort of prioritization of traffic an ISP may be doing without telling you about it. It involved doing a legitimate connection between your own computer and a website and then from a third point on the net inject spoofed packets into your own TCP connection.
Your idea that DNS servers now need to fire off ICMP traffic
I never said that. I said if spoofing is taking place, then the DNS server will receive an ICMP packet. Those ICMP packets are being sent right now, tons of DNS servers across the internet are receiving such ICMP packets right now. I am just suggesting they react to those ICMP packets.
TCP/IP (and ethernet) operate on the assumption that nodes are not lying when they write their source MAC and IP addresses. It is technically possible for a node to do so, but guess what-- it is trivial to implement layer 2 and layer 3 ACLs to stop that crap at the first hop.
There is a major difference. TCP is designed to actually work, even if nodes are spoofing source IP. Ethernet is designed for a LAN where people trust each other. On a LAN it only takes a few packets with spoofed source address to break connectivity, with TCP it takes a significantly larger number of spoofed packets to break connectivity.
Both have been improved since earlier versions, such that they are now better at dealing with spoofing. On Ethernet it requires high-end switches and managing ACLs to prevent problems. With TCP it just required a minor tweak in connection setup to make it even more resilient to spoofed source addresses. And that tweak didn't require any configuration other than an on/off switch to choose if you want the protection or not.
TCP is much better at dealing with spoofed source addresses than raw Ethernet is. I think a few years down the line we'll see a situation where those managed switches are all replaced with routers and a separate/64 network is assigned to each port. At that point MAC spoofing will simply not be an issue anymore.
BCP 38, which does the source address checking where it can be done-- at the first hop controlled by the provider and router of the IP address, the ISP.
Great. I am going to configure that on my bind based DNS server right away.
most hosts do not respond anything to illegitimate UDP packets
I know lots of people block ICMP packets to "protect" their network. Maybe these misguided attempts at protecting networks will stop, if sending those ICMP errors would actually reduce the incoming flow of attack traffic. If it was common practice among DNS servers to take countermeasures, once they receive ICMP errors, then this thread would have looked differently. In that case, it would have started out with explanations about how incompetent the administrators were to put their own network at risk, by blocking legitimate ICMP packets.
even if they did, by the time the DNS server would receive the ICMP error message, it would have already sent out the big response packet, which is causing the DDoS
One packet does not constitute a DDoS attack. And one packet from every single DNS server on the net, would still be a tiny attack.
I already pointed out just looking for ICMP errors is only detecting the illegitimate packet after the fact. The tricky part is what countermeasures to take. And in case we can come up with countermeasures with minimal impact, we could apply those even before knowing if there is an attack. I described one method, which involves sending a truncated reply and using queries received over TCP to signal legitimate clients of the DNS server.
If such countermeasures were used, an attack could not use every DNS server on the net for amplification, but only those, which the victim is already using. That would significantly reduce the effectiveness of the attack.
DNS is stateless.
Not completely. Recursive resolvers need to remember state about all requests in progress. And DNS servers need to be able to receive requests over TCP as well, which is rarely done stateless.
So even if the previous request was illegitimate, will the next request also be?
No, you cannot be sure about that. So you need some reasonable heuristics and a method to ensure that even if you think a query was forged, it can still succeed by switching to TCP.
Assume yes: now you can DoS any server off the Internet by flooding their primary DNS server with illegitimate requests, causing their primary DNS server to assume that also every valid request is illegitimate.
That is obvious. You need to allow for fallback to TCP. This means sending a truncated reply. The truncated reply could be of the same size as the query or smaller, which is sufficient to eliminate the amplification.
: nothing is solved. DNS server may now log that there was an invalid request, but it will still keep sending them out.
Still detecting the problem may be a reasonable first approach. But you still need to apply some countermeasures.
If the problem is to be solved by updating the DNS servers, there would have to be some sensible rate limiting default for UDP requests.
Rate limiting is going to make your DNS server an easier target for DoS attacks. If the rate limiting only applied to blocks detected to be under attack, your DNS server would no longer be such an easy target for a DoS attack.
A new protocol (possibly an extension to DNS as we know it), could provide an even better solution: When the server receives a request, it computes a cookie based on the client IP address. If the request already carried a valid cookie, it can be assumed to not have been forged, and a full reply can be send. If the request did not carry a valid cookie, a short reply is sent with just the cookie. The cookie can be valid for say an hour, and there can be an overlap between successive cookies, such that there is always two valid cookies. This approach would be stateless for the ser
There is no way that DNS over UDP can verify a source address.
You can validate it after the fact. If you receive an ICMP error it was not legitimate. If you can get it to connect over TCP by sending a truncated reply, it was legitimate.
The solution is that all ISPs drop traffic with invalid source addresses before it leaves their network.
This is unrealistic. If you want to design a secure system, you are better off designing it under the assumption, that anybody can spoof source IP addresses. Filtering packets based on source address can be useful in some cases, but it shouldn't bee seen as the solution to DDoS attacks.
Can someone explain how a DNS server can check source address validity?
That's the question I wanted to ask. I can give a partial answer to the question, but no the complete solution implied to exist by the summary.
When the server sends a DNS reply back to a legitimate client, it is not supposed to receive any more packets related to that query. However, if the source address was spoofed, the receiver of the packet will send an ICMP error back indicating the port is closed, or an ICMP error indicating the port is blocked by the firewall, or an ICMP error indicating that the address is unreachable.
You can look for those ICMP errors to find out if your own DNS server is part of the attack with a tcpdump command looking for example like this:
One caveat to remember is, that the ICMP error itself could be spoofed, so some verification would be needed to validate the ICMP error. Also requests spoofing the IP address of a legitimate client of the DNS server means there could be spoofed and legitimate queries arriving from the same source IP address, and just blocking all of those, would introduce a different kind of DoS vulnerability.
So detecting the attack on the DNS server after the fact is easy in principle. But responding to it in a way that doesn't DoS the DNS server itself is tricky. The best I can come up with is force clients to switch to TCP by sending an empty reply with the truncated flag set. That would reduce the amplification factor to one or less, effectively making this DNS server of no use for the attacker. But it would severely reduce the efficiency of DNS lookups on that server, and completely break it for setups, that assumed DNS lookups were always done over UDP.
If there exists a better solution, I'd like to hear about it. Moreover the best solution I can imagine doesn't seem to be implemented by bind. The best I could come up with for bind is:
max-udp-size 512;
Is it going to fire off more packets to that source address (worsening the DDoS) or what?
If you are sending just a single packet to the victim, and that packet is smaller than the DNS request, then the amplification factor is less than one, and the attacker would be better off attacking the target directly. But such a packet needs to be designed to trigger an appropriate response from the DNS client.
One thing you could do when receiving a DNS request from a previously unknown IP address is to always respond with a truncated reply.
If it triggers an ICMP error the request is not legitimate.
If it triggers a request over TCP the request is legitimate, and you can safely send large replies over UDP to that particular IP address until one of them triggers an ICMP error.
If it triggers no response at all, the address you send the reply to is misconfigured in some way. Or the network is congested, which could indicate the address is under attack. In that case you just keep sending truncated replies to that IP address until you know better.
All in all that results in the following simple algorithm.
If you receive a request over UDP and the address is in the cache of known clients send a full reply.
If you receive a request over UDP and the address is not in the cache of known clients send a truncated reply.
If you receive a request over TCP add the address to the cache of known clients.
If you receive an ICMP error (matching a reply you recently send), remove the IP from the cache of known clients. (Optionally implement a counter such that it takes e.g. 10 ICMP errors to remove an entry from the cache.)
As for rate limiting, I looked in the bind manual and found no mention of any way to configure rate limiting. Maybe they used another name for it.
Put at least Stallman, ESR and Torvalds on that plane.
I think it is quite limited what you can get all three of them to agree on. But once they do agree on something, chances are it will be a really good idea.
I'm not sure what you mean by failover. For me failover is active-passive. So one node just sits there and starts the applicaiton when the other node fails.
How about data users are putting into the system? They expect that if they have put data into the system it isn't suddenly lost. So when the primary dies, the secondary must have all the data. But that means you cannot put any new data on the primary unless the secondary is ready to receive it. And how about the situation where communication between the two replicas fail and they declare each other to be dead?
This still requires some sort of coordination before the fact through a secondary communication channel.
That's still better than not being able to communicate at all. Most of the time you only want to chat with people who you have exchanged emails with beforehand anyway.
Of course I have. That's exactly what the last paragraph of my comment was about.
With relatively simple replication techniques you can ensure that your service is running mostly trouble free with minimal operator intervention.
Simple replication techniques can work great if your service is mostly stateless. If your service is stateful then a too simplistic replication causes more problems than it solves. That's why in some cases "good enough" means no replication (except from a regular backup).
All the expensive duplicated hardware in the world won't save you if your clustering software chokes.
I recall a situation where a vendor was trying to sell us an expensive redundant hardware solution. They wanted us to have two replicas of the computer controlling a large machine. But it was deemed too expensive by management on our side. I was actually supportive of that decision for multiple reasons.
The hardware component they wanted to replicate was the most reliable component in the system. It was the only one we had never had a failure on.
The entire computer could easily be replaced with a cold spare with no risk of data loss. The new computer could get up to date data in a few minutes by pulling it from the machine it was controlling.
It wasn't an interactive system with users expecting an answer in seconds. A couple of hours of downtime would go unnoticed by the users.
We already knew the vendor's software had problems keeping data consistent with only a single replica. So how could I trust that they could get a much more complicated solution right?
Finally any real consensus protocol belonged to a much higher layer in the stack than the software running on this computer.
I would like to see this proof. I am not doubting you, I am simply curious about the mathematics involved.
I don't remember which paper the result was in, but I do remember the overall idea of the proof.
The general proof says to handle t failures there must be 3t+1 nodes in total.
It is a proof by contradiction, so initially we assume the nodes can be split into three groups with each node being in exactly one of those three groups. And we assume that any two out of those three groups can reach a consensus without involving the third group. Now we'll prove that under those assumptions, the system breaks down.
So we imagine two completely functional groups out of those three, the network within each group is stable, but the network between them is slow. All the nodes in the third group suffer from a byzantine failure, which cause them to send corrupted messages. Imagine that the third group of failing nodes is still communicating with each of the functional groups, but it sends different information to those two groups. Under those circumstances the failing group along with one group of functional nodes can reach consensus, because we assumed two groups can reach consensus without the third. But at the same time the failing group can reach consensus on a different result with the other group of functional nodes.
In the above partitioning into three groups, we could have t nodes in each group, in which case it is proven that with t failures among 3t nodes we cannot reach consensus. Additionally there exist solutions that will reach consensus with t failures among 3t+1 nodes. They are randomized which means runtime is theoretically unbounded, but the probability that the protocol will take forever is zero. On average it completes quickly. For example the Asynchronous Binary Byzantine Agreement protocol operates in round and has 50% probability of finishing in a given round. If it fails to complete it will run another round and have 50% chance of finishing there. The idea in that protocol is that if there are two candidate results to agree on with roughly the same number of nodes supporting each result, they flip a coin, and try to agree on using the result of the coin flip. Trying to agree on the coin flip can only fail if the coin suggested a result that was behind in the number of nodes supporting it. Hence there is at least 50% chance the coin will land on a side, that leads to agreement.
The byzantine failure model is a bit extreme, but that means protocols designed to work in that model are resilient to extreme failures. The stop dead model on the other hand is a bit unrealistic. Which means protocols designed to work in that model are only proven correct under unrealistic assumptions. They may work in practice most of the time. But the proof of correctness isn't valid in the real world. I don't know if anybody have managed to come up with a sensible model, which lies somewhere between those two.
you only need three computers running a distributed consensus protocol such as Paxos.
Paxos makes assumptions about the behaviour of failing nodes. It assumes a failing node will simply stop sending messages and never send a corrupted message. This assumption is violated if a node for example suffers from memory corruption.
There are two different thresholds for those two models. Allowing for arbitrary failures, the threshold is one third. That means less than one third of the nodes may fail. Once you reach exactly one third, the system breaks down. One out of three is exactly one third, so the system breaks down if one out of three is faulty. One out of four is less than one third, so the system can keep working if one out of four is faulty.
Assuming failing nodes stop responding without ever sending a corrupted message gives a threshold of half. That means less than half the nodes may fail. Once you reach exactly half, the system breaks down. Additionally if a single node has a byzantine failure the system breaks down regardless of the total number of nodes.
won't be able to reach consensus if a network split breaks the cluster into two equal-sized halves
That is true. But if you need to be prepared to handle byzantine failures, it won't help you if the network splits your nodes into approximately equal sized partitions, because you need above two thirds to reach consensus.
One way to see what can go wrong is if you assume you have an odd number of nodes. Then assume one node fails. There will be an even number of nodes left, which you can partition into two equal sized partitions. Assume the network between those two partitions is slow, and the network inside each partition is fast. If the failing node is still able to communicate with each partition, but has a byzantine failure that causes it to give inconsistent answers to the two partitions, it is possible for one partition together with the failing node to reach consensus on one result and the other partition together with the failing node to reach consensus on another result. By the time messages between the two partitions have reached the other side of the slow network, they are too late, because consensus has already been reached.
Excuse me? It's *expensive* yes, hard? Not really.
Most problems gets easier to solve, if you have a lot of money to work with. This one is no exception. But you still need software, which can correctly execute a non-trivial protocol. A single software glitch could still take down the system when the same bug triggers on replicas simultaneously. Redundant systems have blown up due to replicas suffering from the same software glitch.
That means either you need the software to be bug free. Or you let four different teams develop software according to the same spec and hope three of the teams get it right. And even if you have enough money, how do you make sure the people you are hiring are the right people for the task?
Couldn't you theoretically automate what the operator does, and break the 4 computer requirement for failover?
When I said "who you assume can make everything be ok", I was not trying to imply that the assumption was correct. Maybe I should have made that a bit more clear.
The operator is facing a potentially impossible task. The backup may not be entirely up to date, on top of that the faulty computer could potentially have send corrupted data to the backup.
Slashdot Mirror
You are absolutely right, that's how it should have been done. And you can take it in more steps. You don't have to go directly from one to all. Decide how many steps are appropriate and then increase the number of users exponentially between each step.
This sort of approach is standard practice at Google. I don't think Google made the decision to rush this through. I imagine Google was warning about the possibility of something going wrong, but Sky and Yahoo somehow reached an agreement, that Google was simply trying to hold onto the users for as long as possible. So they decided to ignore Google's "warning" and go ahead with a quick transfer of all the users. This is just my imagination of how it might have happened, I don't know if this is exactly how it happened.
A transition period where users are distributed across the two systems requires a special layer of frontends, which know which users are currently on which system. This adds complexity, and complexity increases the risk of bugs. For mails which are arriving via SMTP it is fairly easy to have frontends, which simply delivers copies to both Google and Yahoo. But for anything else it is hard to keep the systems in sync. Doing the entire migration in one go may have been choosen for simplicity. It is possible Sky wanted this approach because it was cheaper.
The documentation from Juniper disagrees with you (first hit when searching for sticky MAC). I do consider documentation from a major vendor of switching hardware to be more trustworthy than comments on slashdot from people I know nothing about.
You can't change your IP in the middle of a TCP connection (though that would be a really nice feature). And some websites will break if you switch IP address in the middle of a session. YouTube streaming has been seen doing that.
If they don't, they will be doing a service to those customers, who do have redundant links. There is nothing in IP mandating packets take the same path in both directions.
I can see you don't know as much about that protocol as I do. But I have written an implementation of the protocol, so I should know how it works.
First of all, the packets I mentioned are not being sent to the relay, they are being sent from the relay. And anybody is allowed to setup such a relay. On a simple home network, you cannot advertise your relay over BGP (and you probably wouldn't want to), but it can service your own LAN without being advertised. So you can have a legitimate relay on your home network. The packets it sends must of course be sent to 6to4 border router, but that can be many hops away, it certainly doesn't have to be within your ISP's network. And the destination could be on any globally routeable IPv4 address (except from 192.88.99.0/24).
DSR will give you much better performance than an equivalent setup without DSR.
It is reasonable to only enable it if customers ask for it. But they can't make the process more complicated than signing up in the first place without pushing some of those customers to other providers.
Doesn't look like it.
Traffic would take a totally different path across the network. A difference in responsiveness across the two paths would be expected, and it does not prove that the ISP is treating traffic differently.
The point isn't whether those are corner cases, the point is how many legitimate use cases I could present right off the top of my head. I think there are plenty more use cases, which I haven't thought about. And maybe some of those use cases will never be fully thought out, simply due to the person who would have come up with a clever idea dropping it because of being restricted by his ISP. Of course the cases I described are not typical, they couldn't be due to many ISPs dropping that sort of traffic. If that traffic hadn't been dropped, such use cases might have been typical.
This is not to say ISPs are doing something wrong by dropping packets they think have a spoofed source address. But neither should we say ISPs are doing something wrong by allowing it. For me it takes only one legitimate use case to defend those ISPs who do permit it. The root of the problem is not spoofing, but amplification. (I have experienced a case that would make a factor 100 amplification look harmless, unfortunately I am not allowed to go into details about that episode.)
Completely eliminating problems with MAC spoofing is worth something. It would mean even if a vendor by mistake assigned same MAC to two devices, you could plug them both into the same router, and it would just work.
My calculations says the opposite. But it is really a minor difference, because it is only for the initial neighbor discovery, there will be a difference. Once receiving MAC addresses are known, the actual payload traffic takes the exact same physical path, hence no change in latency. For the neighbor discovery it is more likely for each node having an entry for the router in cache already and the router having entries for the devices, than it is for the devices having entries for each other. So the probability that you need to wait for a neighbor discovery is smaller. Moreover the latency of that discovery will be smaller, as it will need to go over only a single Ethernet link, rather than the two links needed in case of doing it between two devices each connected to a switch.
This is already being considered by the IETF.
Which is a problem only if you are doing this using public IPv4 addresses. But the networks we are talking about here aren't using public IPv4 addresses in the first place.
Unless you want the introduced security it offers to your LAN. Some attacks works only within the same segment. Putting every port on a separate segment means you only need to protect the router against such attacks and not every single device.
Those are not the only sorts of attacks that you can perform within a segment. And learning MAC addresses doesn't make sure your devices will actually be able to communicate in case the vendor screwed up and manufactured two with identical MAC addresses. Putting each port on a separate segment would solve that problem.
With the proper chips, routing can be done at wire-speed as well. It is not even much more complicated to do so. Of course what I mentioned won't happen until the devices are being manufactured with chips capable of doing it.
Finding out which port a packet must leave on based on the destination IP address is the defining feature of a
Of course it won't work if the line is completely down, or if it for some other reason is unable to deliver traffic to you. But if the line can deliver packets to you, and only the outgoing path is having a problem, then it can indeed help. He cannot give you any percentage of what kind of reliability improvement it could give, since I haven't had the opportunity to measure such a setup.
I was not trying to imply you could do a TCP session working with a spoofed IP, quite the contrary. My point was the packets with spoofed IP address does not immediately break legitimate traffic on that IP. In fact it is quite difficult to break established TCP connections, if you are not on the path it flows through, even if you have access to spoof an IP address. That a server receiving incoming TCP connections can be fairly confident the client IP is authentic, is an added bonus.
On a switched network with non-managed switches, spoofing the MAC address of another computer may be enough to kick that other computer off the network. You need to use managed switches and configure ACLs on those to stop that sort of attack.
If it performs routing, then it is a router. Hence if that's the way we are heading, I was right saying the switches will be replaced by routers.
And I expect that to be taken one step further. There already exist chips, which can be either a switch or a router depending on how you configure them. And if you just configure such a chip to act as a router with every port being on a different segment, then you don't need to worry about MAC spoofing.
Somebody did already, they just didn't do a very good job at it. EDNS made this sort of attack 8 times more effective than it was before EDNS was introduced.
That is a very sensible approach. It does however require a little bit more management for the ISP to do this, and they will likely charge extra for that. Customers who have a choice may decide to go with an ISP that permit it by default, because it is simpler than figuring out how to get it turned on, and probably cheaper too.
If that ISP decides to turn it off, and they actually have customers relying on it, they'll suddenly have to deal with support calls.
If it actually had a cost to the ISP, who was permitting spoofed source addresses, the situation might look different. But as it is, tracking down the source of packets with spoofed source is a bit tricky.
It could be delivered through a different path. For example you might have redundant internet connections through different providers. It could also be that one of the two paths is using VPN and the other is not.
There are also special addresses, which are valid as source address, even if they are not assigned to you specifically. For example, I could send packets through my ISP with source address 192.88.99.1, which would be perfectly legitimate and not perform any IP spoofing at all. But even though the packet is perfectly legitimate, and the reply would indeed be routed back to me, that probably won't work, because my provider is filtering and didn't know about 192.88.99.1, when configuring the filters.
If we look at hosting companies, there is another reason why a customer may need to send packets with apparently spoofed source addresses. It may be that the customer is simply doing load balancing across multiple data centers, and are using a DSR setup.
At some point I watched a talk mentioning another legitimate use case for apparently spoofed source address. It was for investigating what sort of prioritization of traffic an ISP may be doing without telling you about it. It involved doing a legitimate connection between your own computer and a website and then from a third point on the net inject spoofed packets into your own TCP connection.
I never said that. I said if spoofing is taking place, then the DNS server will receive an ICMP packet. Those ICMP packets are being sent right now, tons of DNS servers across the internet are receiving such ICMP packets right now. I am just suggesting they react to those ICMP packets.
There is a major difference. TCP is designed to actually work, even if nodes are spoofing source IP. Ethernet is designed for a LAN where people trust each other. On a LAN it only takes a few packets with spoofed source address to break connectivity, with TCP it takes a significantly larger number of spoofed packets to break connectivity.
/64 network is assigned to each port. At that point MAC spoofing will simply not be an issue anymore.
Both have been improved since earlier versions, such that they are now better at dealing with spoofing. On Ethernet it requires high-end switches and managing ACLs to prevent problems. With TCP it just required a minor tweak in connection setup to make it even more resilient to spoofed source addresses. And that tweak didn't require any configuration other than an on/off switch to choose if you want the protection or not.
TCP is much better at dealing with spoofed source addresses than raw Ethernet is. I think a few years down the line we'll see a situation where those managed switches are all replaced with routers and a separate
Great. I am going to configure that on my bind based DNS server right away.
I know lots of people block ICMP packets to "protect" their network. Maybe these misguided attempts at protecting networks will stop, if sending those ICMP errors would actually reduce the incoming flow of attack traffic. If it was common practice among DNS servers to take countermeasures, once they receive ICMP errors, then this thread would have looked differently. In that case, it would have started out with explanations about how incompetent the administrators were to put their own network at risk, by blocking legitimate ICMP packets.
One packet does not constitute a DDoS attack. And one packet from every single DNS server on the net, would still be a tiny attack.
I already pointed out just looking for ICMP errors is only detecting the illegitimate packet after the fact. The tricky part is what countermeasures to take. And in case we can come up with countermeasures with minimal impact, we could apply those even before knowing if there is an attack. I described one method, which involves sending a truncated reply and using queries received over TCP to signal legitimate clients of the DNS server.
If such countermeasures were used, an attack could not use every DNS server on the net for amplification, but only those, which the victim is already using. That would significantly reduce the effectiveness of the attack.
Not completely. Recursive resolvers need to remember state about all requests in progress. And DNS servers need to be able to receive requests over TCP as well, which is rarely done stateless.
No, you cannot be sure about that. So you need some reasonable heuristics and a method to ensure that even if you think a query was forged, it can still succeed by switching to TCP.
That is obvious. You need to allow for fallback to TCP. This means sending a truncated reply. The truncated reply could be of the same size as the query or smaller, which is sufficient to eliminate the amplification.
Still detecting the problem may be a reasonable first approach. But you still need to apply some countermeasures.
Rate limiting is going to make your DNS server an easier target for DoS attacks. If the rate limiting only applied to blocks detected to be under attack, your DNS server would no longer be such an easy target for a DoS attack.
A new protocol (possibly an extension to DNS as we know it), could provide an even better solution: When the server receives a request, it computes a cookie based on the client IP address. If the request already carried a valid cookie, it can be assumed to not have been forged, and a full reply can be send. If the request did not carry a valid cookie, a short reply is sent with just the cookie. The cookie can be valid for say an hour, and there can be an overlap between successive cookies, such that there is always two valid cookies. This approach would be stateless for the ser
You can validate it after the fact. If you receive an ICMP error it was not legitimate. If you can get it to connect over TCP by sending a truncated reply, it was legitimate.
This is unrealistic. If you want to design a secure system, you are better off designing it under the assumption, that anybody can spoof source IP addresses. Filtering packets based on source address can be useful in some cases, but it shouldn't bee seen as the solution to DDoS attacks.
That's the question I wanted to ask. I can give a partial answer to the question, but no the complete solution implied to exist by the summary.
When the server sends a DNS reply back to a legitimate client, it is not supposed to receive any more packets related to that query. However, if the source address was spoofed, the receiver of the packet will send an ICMP error back indicating the port is closed, or an ICMP error indicating the port is blocked by the firewall, or an ICMP error indicating that the address is unreachable.
You can look for those ICMP errors to find out if your own DNS server is part of the attack with a tcpdump command looking for example like this:
One caveat to remember is, that the ICMP error itself could be spoofed, so some verification would be needed to validate the ICMP error. Also requests spoofing the IP address of a legitimate client of the DNS server means there could be spoofed and legitimate queries arriving from the same source IP address, and just blocking all of those, would introduce a different kind of DoS vulnerability.
So detecting the attack on the DNS server after the fact is easy in principle. But responding to it in a way that doesn't DoS the DNS server itself is tricky. The best I can come up with is force clients to switch to TCP by sending an empty reply with the truncated flag set. That would reduce the amplification factor to one or less, effectively making this DNS server of no use for the attacker. But it would severely reduce the efficiency of DNS lookups on that server, and completely break it for setups, that assumed DNS lookups were always done over UDP.
If there exists a better solution, I'd like to hear about it. Moreover the best solution I can imagine doesn't seem to be implemented by bind. The best I could come up with for bind is:
If you are sending just a single packet to the victim, and that packet is smaller than the DNS request, then the amplification factor is less than one, and the attacker would be better off attacking the target directly. But such a packet needs to be designed to trigger an appropriate response from the DNS client.
One thing you could do when receiving a DNS request from a previously unknown IP address is to always respond with a truncated reply.
All in all that results in the following simple algorithm.
As for rate limiting, I looked in the bind manual and found no mention of any way to configure rate limiting. Maybe they used another name for it.
I think it is quite limited what you can get all three of them to agree on. But once they do agree on something, chances are it will be a really good idea.
Can they do something about the high pitched ringing tone inside my head?
How about data users are putting into the system? They expect that if they have put data into the system it isn't suddenly lost. So when the primary dies, the secondary must have all the data. But that means you cannot put any new data on the primary unless the secondary is ready to receive it. And how about the situation where communication between the two replicas fail and they declare each other to be dead?
That's still better than not being able to communicate at all. Most of the time you only want to chat with people who you have exchanged emails with beforehand anyway.
True. If your service is stateless, it becomes much simpler. The complexity I was referring to applies to stateful services.
If it is done in a sensible way, you will still be able to communicate, as long as both parties invite each other.
Of course I have. That's exactly what the last paragraph of my comment was about.
Simple replication techniques can work great if your service is mostly stateless. If your service is stateful then a too simplistic replication causes more problems than it solves. That's why in some cases "good enough" means no replication (except from a regular backup).
See http://slashdot.org/comments.pl?sid=3547941&cid=43180375
I recall a situation where a vendor was trying to sell us an expensive redundant hardware solution. They wanted us to have two replicas of the computer controlling a large machine. But it was deemed too expensive by management on our side. I was actually supportive of that decision for multiple reasons.
I don't remember which paper the result was in, but I do remember the overall idea of the proof.
The general proof says to handle t failures there must be 3t+1 nodes in total.
It is a proof by contradiction, so initially we assume the nodes can be split into three groups with each node being in exactly one of those three groups. And we assume that any two out of those three groups can reach a consensus without involving the third group. Now we'll prove that under those assumptions, the system breaks down.
So we imagine two completely functional groups out of those three, the network within each group is stable, but the network between them is slow. All the nodes in the third group suffer from a byzantine failure, which cause them to send corrupted messages. Imagine that the third group of failing nodes is still communicating with each of the functional groups, but it sends different information to those two groups. Under those circumstances the failing group along with one group of functional nodes can reach consensus, because we assumed two groups can reach consensus without the third. But at the same time the failing group can reach consensus on a different result with the other group of functional nodes.
In the above partitioning into three groups, we could have t nodes in each group, in which case it is proven that with t failures among 3t nodes we cannot reach consensus. Additionally there exist solutions that will reach consensus with t failures among 3t+1 nodes. They are randomized which means runtime is theoretically unbounded, but the probability that the protocol will take forever is zero. On average it completes quickly. For example the Asynchronous Binary Byzantine Agreement protocol operates in round and has 50% probability of finishing in a given round. If it fails to complete it will run another round and have 50% chance of finishing there. The idea in that protocol is that if there are two candidate results to agree on with roughly the same number of nodes supporting each result, they flip a coin, and try to agree on using the result of the coin flip. Trying to agree on the coin flip can only fail if the coin suggested a result that was behind in the number of nodes supporting it. Hence there is at least 50% chance the coin will land on a side, that leads to agreement.
The byzantine failure model is a bit extreme, but that means protocols designed to work in that model are resilient to extreme failures. The stop dead model on the other hand is a bit unrealistic. Which means protocols designed to work in that model are only proven correct under unrealistic assumptions. They may work in practice most of the time. But the proof of correctness isn't valid in the real world. I don't know if anybody have managed to come up with a sensible model, which lies somewhere between those two.
Paxos makes assumptions about the behaviour of failing nodes. It assumes a failing node will simply stop sending messages and never send a corrupted message. This assumption is violated if a node for example suffers from memory corruption.
There are two different thresholds for those two models. Allowing for arbitrary failures, the threshold is one third. That means less than one third of the nodes may fail. Once you reach exactly one third, the system breaks down. One out of three is exactly one third, so the system breaks down if one out of three is faulty. One out of four is less than one third, so the system can keep working if one out of four is faulty.
Assuming failing nodes stop responding without ever sending a corrupted message gives a threshold of half. That means less than half the nodes may fail. Once you reach exactly half, the system breaks down. Additionally if a single node has a byzantine failure the system breaks down regardless of the total number of nodes.
That is true. But if you need to be prepared to handle byzantine failures, it won't help you if the network splits your nodes into approximately equal sized partitions, because you need above two thirds to reach consensus.
One way to see what can go wrong is if you assume you have an odd number of nodes. Then assume one node fails. There will be an even number of nodes left, which you can partition into two equal sized partitions. Assume the network between those two partitions is slow, and the network inside each partition is fast. If the failing node is still able to communicate with each partition, but has a byzantine failure that causes it to give inconsistent answers to the two partitions, it is possible for one partition together with the failing node to reach consensus on one result and the other partition together with the failing node to reach consensus on another result. By the time messages between the two partitions have reached the other side of the slow network, they are too late, because consensus has already been reached.
Most problems gets easier to solve, if you have a lot of money to work with. This one is no exception. But you still need software, which can correctly execute a non-trivial protocol. A single software glitch could still take down the system when the same bug triggers on replicas simultaneously. Redundant systems have blown up due to replicas suffering from the same software glitch.
That means either you need the software to be bug free. Or you let four different teams develop software according to the same spec and hope three of the teams get it right. And even if you have enough money, how do you make sure the people you are hiring are the right people for the task?
When I said "who you assume can make everything be ok", I was not trying to imply that the assumption was correct. Maybe I should have made that a bit more clear.
The operator is facing a potentially impossible task. The backup may not be entirely up to date, on top of that the faulty computer could potentially have send corrupted data to the backup.