How to Work Around Broken Port-80 Routing?

Dr. Zowie writes "My ISP places an opaque (intended to be transparent) web proxy between me and the rest of the world. It is causing me problems due to misconfiguration or misdesign. My question is twofold. On the micro level, what can I do in the short term to work around the broken routing (in the long term, I switch ISPs if it's not fixed)? On the macro level, what can we as a community do to prevent breakage of the net on a global scale by poorly designed routing hacks?"

Dr. Zowie continues: "I use a regional ISP with otherwise-very-good policies. However, they seem to be intercepting anything that comes from my home net on port 80, so that they can ``transparently'' cache web requests based on the payload of those packets. The proxy seems to work rather well in most cases: I never noticed it until I started using OpenNIC. Then I found that some web pages that should have resolved OK through the OpenNIC system failed even though routing on different ports worked OK.

"I did some experimentation using ``telnet'' on port 80 directly, and found that packets are being routed based only on the payload regardless of the original destination address: I can (for example) retrieve the Slashdot front page by using ``telnet www.google.com 80'' and asking for "http://www.slashdot.org http/1.1". The tech support folks seem to be stonewalling me: the main contact tells me that the behavior is "not broken" even though it clearly violates RFC 1812, the standard set of rules for IP routing.

"The practice of ``transparent'' proxy routing seems to be growing more widespread. It appears to break the internet standard in a way that works for most folks for now, but that breaks port 80 usage in general. Looking ahead, this breakage seems like a growing nightmare waiting to happen. At the very least, I expect more instances of my particular problem to appear as folks give up on the corporate hegemony of ICANN. More insidiously, transparent proxy routers break the layered nature of the internet protocol and restrict the flexibility that made it work in the first place. One would hope that such proxies would at least act like routers when the fancier proxying fails, but at least my ISP's doesn't. What about your ISP's?"

Comments

Tunneling

[Matthaeus]

I recently had this problem with my university account...They route all resnet web traffic through an old 386 proxy server that can't handle the load. Find a free proxy out there and SSH tunnel to it. I'm sure there are more elegant means of getting through a poorly configured proxy, but this'll work as a quick fix.

same problem

[babycakes]

We had pretty much the exact same problem with our ISP, in that if we sent HTTP requests out without any proxy configuration, they would often take a couple of times to get through, since our ISP's transparent proxying didn't work. However, on setting the browser's proxy settings to the proxy itself, this seemed to solve the problem since it would ask the proxy directly.

Don't ask me why :)

Education

[radoni]

At my highschool, the current system for blocking webpages was introduced as a means to cache commonly used pages and make the District 225 intranet faster. The superintendent and members of the district board know very little about computers, so naturally it is approved. After the Columbine incident, a new feature was tacked on that blocked certain objectionable web sites. The recent WTC attack caused even more areas of the net to be restricted. Today, when i want to search "terrorism" for a paper on the war afghanistan, my results are blocked. Teachers have informed us that we must use the one non-blocked computer in the tech room, or do research at home.

my friend set up an anonymous web surfing proxy at his home computer, and using this i can get whatever i want.

there are publically available anonymous port-80 proxies still around.

Re:Education

[Ryan Amos]

Obviously you've never dealt with a public school system. Refuse using the machines? You get written up and sent to the principal for disobeying the teacher. Tell the board anything? They don't listen to teachers or parents, let alone students. Teachers are often just as powerless as the students in administration matters. Unfortunately, public schools often operate on the assumption that the people on top are ALWAYS right.

The school board of any decently large school district is generally disinterested in actually educating students or making things work well, most school board members are just there as a springboard for higher political office. They generally don't give a fuck about the students or education, so they lower test standards and claim that test scores have improved. If it makes them look good to "protect our children from evil" by blocking out these sites, they'll do it. School systems don't operate like normal organizations; the students' opinion carries next to no weight at all, as "it is up to us adults who know better to protect the students from what they don't know." Total BS, but that's the way public school works.

My Experience on ISP with faulty service

[Jucius Maximus]

I used to have an ISP that, although they allowed you to have your own site (on their webspace,) loading the site was just damn SLOW for anyone who tried. It was much faster if the pages were hosted somewhere on another continent compared to an ISP with a server in the same city.

The thing is, they probably won't listen to problems like this, or your proxy issue in most cases. But I found a way to make them listen to you:

Phone them up saying that you want to cancel the service. Mention something about their web hosting being broken. They will probably say that they will have a management person phone you back to confirm the process.

When they do phone back, for me, the call was like "Hello, there was a call eariler about a slow connection?" And at this point you have someone on the line who is interested in helping you, has power in the organisation to really fix things (because they're management or a senior tech) and they want to get your issue fixed to they don't lose your business. And THIS is when you really try to explain what's going on.

This was my experience. Perhaps it will work for you.

ISPs required by law to block port 80 in Singapore

[tangent3]

Here in Singapore, ISPs are required by law to block port 80, forcing all outgoing http requests to go through a proxy server (which filters out webpages which are deemed unsuitable for Singaporeans to view, including www.playboy.com), or to have a transparent proxy server blocking out such requests.

This has caused me many problems before, when my IP gets determined wrongly by the remote site (which naturally thinks takes the proxy server's IP for my IP address). Some applications don't like the transparent proxy either, for example Frontpage Extension (not my choice to use!), and an autopatching program which refused to download the latest version of a file, insisting on downloading only the file cached in the proxy server until the cache gets flushed.

The only real method of bypassing the proxy is to use another proxy server (since 8080 isn't blocked) outside the ISP's network. This tends to be really slow though.

I guess I have to live with this until the government one day realises that proxy servers cannot stop the people from viewing pr0n, and it's probably not worth maintaining the proxy servers to meet the demands of all the net users in Singapore, not to mention maintaining the list of sites to block.

What brand of transparent proxy is it?

[billstewart]

Do you know what brand of attempting-to-be-transparent proxy cache server they're using? Proxy caching is an important performance enhancer for ISPs, corproate firewalls, and other bottleneck network environments, and "transparent" proxies are less trouble for the ISP and for the users as well (especially since many users wouldn't bother configuring their browsers for them unless either they're pre-configured by the ISP or forced to use the proxy by firewall rules that block non-proxy access.)

Of course, the problem with transparent servers is when they're not, and your ISP seems to have one that isn't. Is it possible to find out what kind it is, either by telnetting to the thing and looking at headers or by asking the ISP, and can you do bug reports to the vendor to get them to fix their product?

AOL ignores ports

[Anonymous Coward]

AOL's transparent proxy is a little worse. It ignores the port and proxies anything that looks like HTTP. Of course, they deny having a transparent proxy, but I was able to watch packets leaving our network headed for AOL and then watch altered packets come back from AOL.

I stumbled across this when their proxy had some trouble with the cookies we were using and suddenly no one on AOL could use our service. A few minutes later they could again. Then they could not. During this time, I was running a packet logger on the outgoing traffic from our server and on the incoming traffic to a workstation I had connect to AOL. Everything worked find until the server sent the cookie. Then AOL suddenly stopped sending more packets. This occured on every port I tried, even ports reserved for other services.

Re:corrections, suggestions, etc

[Phroggy]

Tech support people are lazy, however, in some cases, and may just opt to cancel you.

Au contraire. Tech support people are tired of listening to customers whine about problems that tech support people cannot fix. If customers have unreasonable expectations, and refuse to listen to us, it's far better for the company if they just cancel service and go elsewhere (becoming somebody else's problem).

Also, non-chalance about canceling service is sometimes the best way to make customers understand that we really are doing our best to help them, and we're not just blowing them off. Sounds weird, but here's an example:

Customer has a problem with their DSL service. We've identified that the problem lies with the phone company. Phone company has given us a commit date of Tuesday by end of business day for repair to be complete. For whatever reason, the customer feels like they've been dragged around, and their service isn't getting fixed. Customer says if they're not up and running by 9:00am Monday morning, they're cancelling service.

Customer expects us to bend over backwards to get them up and running by 9:00am Monday morning. We can't. There is absolutely nothing we can do. It's out of our hands. Customer needs to understand this. Customer will have the same problem at any competing DSL ISP, but we're the ones who have identified the problem and are getting it fixed.

We respond by repeating to the customer that we have been given a commit time of Tuesday by end of business day, but that we cannot guarantee that the issue will be resolved by then. We then offer to the customer that if this is unacceptible and they'd prefer to cancel service, although we'd hate to lose them as a customer, we'd be more than happy to transfer them to someone who can take care of that.

This has the effect of making it clear to the customer that we really mean what we say. Usually, they shut up, keep their account, and let us do our jobs. Often, they'll ask to be transferred to get the account cancelled, then hang up during the transfer.

The alternative is to offer the customer incentives to try to convince them to stay with us, such as offering a free month of service, or a credit on their account. This costs us money, and gains nothing - if the customer has the expectation that we're willing to give him free service, he'll try to take advantage of it in the future. Far too many ISPs have failed for this very reason.

At the last few ISPs I've worked for, nearly all my coworkers have been genuinely interested in helping the customer, and we've been fortunate to have management that allows us to do so. I understand that at some companies this is not the case; those are obviously the ones to avoid.

Sorry for ranting. Getting back on track: ultimatums like "if you don't fix this problem, I'll cancel my service" sometimes are a good idea. That will tell you whether or not you can get the issue resolved. Be prepared to actually cancel, because if they can't resolve the issue, that's what will happen. If they can but just don't want to, threatening to cancel may just be the incentive they need to get it done.

Re:Hold on here!

[Skapare]

If the user configured his browser to use a specific proxy, then I would agree with you regarding RFC2068. The client in essence is delegating DNS responsibility to the proxy server. However, what is happening here is called transparent proxy. There is no DNS delegation taking place. And RFC2068 requires that semantic transparency be preserved (although it does not seem to differentiate types of proxies). It says:

semantically transparent
A cache behaves in a "semantically transparent" manner, with
respect to a particular response, when its use affects neither the
requesting client nor the origin server, except to improve
performance. When a cache is semantically transparent, the client
receives exactly the same response (except for hop-by-hop headers)
that it would have received had its request been handled directly
by the origin server.

In this case the origin server would have delivered a web page (I actually tried it and it works fine for me), and so the proxy has the responsibility to deliver the same thing. In that, it seems, it failed.

Have you tried asking them?

[wizman]

I am the network admin for a wireless isp that does transparent cacheing. If a user asks us to turn it off, we can disable it for their IP.

For more than 99% of our users, they don't know what routing or cacheing is, much less that it's happening. For those that actually have issues with the proxy it's a quick modification to our ipchains rules. So far we've only had 2 such requests. Also, we disable the cacheing for business class users by default.

I would hope that you would ask them to disable their transparent cacheing for you before doing something as rash as dropping them. It's my bet that most of their other users do not have this issue, and they may not even be aware that it is causing problems for you.