Slashdot Mirror
Google Caught in Comcast Traffic Filtering?
marcan writes "Comcast users are reporting 'connection reset' errors while loading Google. The problem seems to have been coming and going over the past few days, and often disappears only to return a few minutes later. Apparently the problem only affects some of Google's IPs and services. Analysis of the PCAP packet dumps reveals several injected fake RSTs, which are very similar to the ones seen coming from the Great Firewall of China [PDF]. Did Google somehow get caught up in one of Comcast's blacklists, or are the heuristics flagging Google as a file-sharer due to the heavy traffic?"
After all, doesn't Google host more copyrighted content than any other person/company in the world? ;)
Is the title clear enough? I can't imagine any judge or jury saying Comcast is allowed to impersonate Google and tell Comcast customers they're not allowed to use Google's services or that Google's services are overwhelmed and shutting down connections. That's essentially what forged, fraudulent RST packets from a MITM attack are doing. That can't possibly be considered a legitimate business practice in court.
There's a lot of guesswork here about what providers may or may not be doing; are there any applications for actually testing ISPs? Such testing apps would discover traffic shaping, port filtering, connectivity, and other traffic modifications by the ISP. Something like a bandwidth tester on steroids.
The problems with a fake RST detector are two-fold. The RST bits are being set on TCP traffic sent in both directions on a connection, so even if you ignore RST teardowns, the other side will tear down the connection. What Sandvine boxes do is just flip the RST bits on TCP packets flowing through them, so the sequence numbers will appear correct in the connection tracking table because the TCP packet is a valid one from the other side of the connection.
If Comcast truly is using Sandvine boxes, then this could be a network controller station with the preset examples still in place. The Sandvine sales presentation shows how to load up the system with all the prefixes from AS36561, and then interfere with a tiny percentage of TCP traffic after the first few hundred packets are transferred. What this does is provide a way of denying they are completely blocking those packets, but will blow away any connection hoping to do streaming video or cruise around on a web page heavy in graphic content like a mapping function.
The business model after installing Sandvine boxes is to then extort regular payments from large content providers to allow access to their network. Comcast, SBC/ATT and a few other monopolistic ISPs would like to see both sides of a connection pay for traffic in both directions, not the current economic model where each side pays for their own access or transit.
What Sandvine boxes do is break the end-to-end model of the internet. Even a tiny percentage of broken connections will put an end to all the cool applications everyone is currently enjoying. Streaming video and audio sessions, VoIP calls, file downloads, p2p exchanges, search engines, mapping and geolocation, and heavy web content sessions like social networking sites. The only traffic that can survive this kind of interference are from applications that make repeated attempts at connection in case of unexpected interruptions, like SMTP.
P2P protocol designers are pretty agile and clever. In the face of regular faked TCP RST bits on a connection, they'll evolve the protocol to make shorter connections, and to make repeated attempts to reconnect when an unexpected RST is received. Expect tuning "knobs" in clients very soon now, on how resilient to make the connections or how many bytes to transfer before tearing down and rebuilding the connection. There could also be a way to limit the numbers of attempted connections so as to fly under the radar of systems like this. I can open any bittorrent client with a single popular file, and see over 1000 completed TCP connections within 2 to 3 minutes. Limiting the number of new connections per minute could throw a spanner in Sandvine's current design.
the AC
Hemos is like...sci-fi fans;he thinks technology is cool, but he hasn't bothered to understand the science it's based on
Nah, the basic problem is that the bigger the company, the higher the density of PHBs. Once you get to a certain concentration, you hit stupidity critical mass. From the outside it looks like malice, but it's really just highly focused incompetence.
Oh, if that were really the only problem.
There are two kinds of big mistakes you can make: those that are big for a company your size, and those that are just plain big. In a big company with lots of customers, small mistakes are multiplied by volume into just plain big mistakes. If you've got gross revenues of a million dollars, a mistake with a potential $100,000 impact is big for your business, but not that big. You can survive it, you can reestablish credibility with your customers (whom you know face to face) by personally eating a helping of crow in front of each and every one. If you're in a company a 100x as big, you're talking maybe a $10M impact that if laid to the account of any individual employee is a disaster beyond that individual's ability to make right.
That's why large companies can develop a special kind of stupidity, preferring a status quo that is certainly wrong to any alternative that is only probably right. Individuals protect themselves using exactly the same strategy that schooling fish employ. Any decision has to have so many fingerprints on it that firing the people who can be tied to a mistake is like cutting off your right arm. That's why big defense contractors are probably the most bureaucratic organizations on the planet. Ordinary mortals have to make decisions that can have impacts measured in hundreds of millions of dollars. In any such situation, you obviously need a form of collective responsibility, the question is what form it takes. It's all to easy to develop an organization that protects individuals by being unable to detect and respond to most problems. We didn't know about it, if we had we probably couldn't do anything about it, and if we could have, it wasn't my job.
The problem is not that a typical PHB is necessarily stupid. The problem is that organizations are built in a way that rewards people for acting in a stupid way. But stupidity is all too common. Even stupid people can manage to be cunning in bad organizations, because they are problems in an organization built around willful blindness to problems. It's more of a challenge for intelligent people I suppose, because it's hard for people with imagination to find much satisfaction in what it takes to get ahead in these places. It has even been suggested that sociopaths make good managers, which I doubt. But I can well believe that feigned stupidity is better in some cases than the real thing.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.