Slashdot Mirror
How to Work Around Broken Port-80 Routing?
Dr. Zowie continues: "I use a regional ISP with otherwise-very-good policies. However, they seem to be intercepting
anything that comes from my home net on port 80, so that they can
``transparently'' cache web requests based on the payload of those
packets. The proxy seems to work rather well in most cases: I
never noticed it until I started using OpenNIC. Then I found that some web pages that should have
resolved OK through the OpenNIC system failed even though routing on
different ports worked OK.
"I did some experimentation using ``telnet'' on port 80
directly, and found that packets are being routed based only on
the payload regardless of the original destination address: I can (for
example) retrieve the Slashdot front page by using ``telnet
www.google.com 80'' and asking for "http://www.slashdot.org
http/1.1". The tech support folks seem to be stonewalling me: the
main contact tells me that the behavior is "not broken" even though it
clearly violates RFC
1812, the standard set of rules for IP routing.
"The practice of ``transparent'' proxy routing seems to be growing
more widespread. It appears to break the internet standard in a way
that works for most folks for now, but that breaks port 80 usage in general. Looking ahead, this breakage seems
like a growing nightmare waiting to happen. At the very least, I
expect more instances of my particular problem to appear as folks give up on the corporate hegemony of ICANN. More insidiously, transparent
proxy routers break the layered nature of the internet protocol and
restrict the flexibility that made it work in the first place. One would
hope that such proxies would at least act like routers when the fancier
proxying fails, but at least my ISP's doesn't. What about your ISP's?"
You can use netcat to route your own port 80 traffic. Simply get a good UNIX shell account, and configure your router to direct to that. It becomes a real version of what you would be trying to do. However, I would bitch like crazy if my ISP did anything like that to me. If I want to connect to port 80 on something, I would want to be connecting to such port 80. Any fiddling with it would sure make me drop that ISP in an instant.
samrolken
Or, you could use your own proxy server, like Squid for UNIX or AnalogX Proxy for Win32. You might try something like the port + 65536 rule. Port 80 becomes port 65616 or something (That may not be precise), and that would confuse your router, but still be port 80. I used a similar trick to get around similar proxying at school.
samrolken
I recently had this problem with my university account...They route all resnet web traffic through an old 386 proxy server that can't handle the load. Find a free proxy out there and SSH tunnel to it. I'm sure there are more elegant means of getting through a poorly configured proxy, but this'll work as a quick fix.
The proper policy for an isp should be using an optional proxy.
:)
:)
That's how my connection works, and it works just beautifully - since I can set it to use the proxy for some sites, but not other
Perfect for all uses. and the average user has no idea
find a friend who has a colocated server or dsl connection.
then use that machine as a web proxy, or set up an ipsec tunnel to that machine and route your port 80 traffic through that tunnel.
bgphints - internet routing news, hints and ti
I should have posted all this in one comment... oh well...
n et /Proxies/Free/?tc=1
You could also use a third party proxy server. You can find gobs of them here:
http://tools.rosinstrument.com/proxy/
and here:
http://directory.google.com/Top/Computers/Inter
samrolken
many ISPs do this, just use a different port. I use
port 8001
The real problem is that you're probably using port 80 for something other than what it's explicit purpose. Port 80 is the domain of HTTP, which has its own set of rules.
As long as you're using HTTP, the "payload" is all that matters! A proxy is certainly not required to route anything anywhere! It's your fault--not your ISP's--that you choose to disregard the other relevant standards. If you would embrace the DNS standard, HTTP standard and the routing standard, you'd have no troubles.
Without some "outside" help, there isn't much you can do. In the long run however, it would help if everyone tried to use protocols which are opaque to the transport: SSL, IPSec, etc.
Onenet is the internet "service" provider to most state agencies within Oklahoma, including Oklahoma State University, where I am currently working on a BSEE. Neglecting Onenet's other issues (AOL's netadmins could do a better job than Onenet's), they have a "transparent" web cache proxy. More often than not, errors fetching a web page come not from the browser or the site itself as they should, but from the proxy. DNS errors from the proxy are not uncommon. As for switching ISPs, I can't, which really sucks. But for what I can reach on the net, I'm still getting ultra-cheap broadband :P.
I pledge allegiance to the flag...
of the Corporate States of America...
How's running your own proxy supposed to help? The connection to the server on the other side is to port 80, so it will be channeled through whatever proxy his ISP has in place, no matter if it's coming from your webbrowser or your proxy. Besides, while that would be against every convention, port 80 could be used for something very different from http, in which case any web proxy would break it.
We had pretty much the exact same problem with our ISP, in that if we sent HTTP requests out without any proxy configuration, they would often take a couple of times to get through, since our ISP's transparent proxying didn't work. However, on setting the browser's proxy settings to the proxy itself, this seemed to solve the problem since it would ask the proxy directly.
:)
Don't ask me why
At my highschool, the current system for blocking webpages was introduced as a means to cache commonly used pages and make the District 225 intranet faster. The superintendent and members of the district board know very little about computers, so naturally it is approved. After the Columbine incident, a new feature was tacked on that blocked certain objectionable web sites. The recent WTC attack caused even more areas of the net to be restricted. Today, when i want to search "terrorism" for a paper on the war afghanistan, my results are blocked. Teachers have informed us that we must use the one non-blocked computer in the tech room, or do research at home.
my friend set up an anonymous web surfing proxy at his home computer, and using this i can get whatever i want.
there are publically available anonymous port-80 proxies still around.
SIGERR: laziness exceeds quota
I know my ISP's AUP doesn't allow ANY servers of ANY type (which is rediculous, but I know for a fact that whining will get me nowhere with them). One of the ways they do this is to actually block anything coming in on port 80 to block an http server. Of course, I just change the port, but it could very well be that your isp just doesn't want you running a server and is trying to find an automated solution to stop most people.
that's why I suggested adding 80 to 2^16 and setting your proxy to connect at that port. It's the same port, the auto-proxy-router thing just wouldn't see it as such.
samrolken
The thing is, they probably won't listen to problems like this, or your proxy issue in most cases. But I found a way to make them listen to you:
Phone them up saying that you want to cancel the service. Mention something about their web hosting being broken. They will probably say that they will have a management person phone you back to confirm the process.
When they do phone back, for me, the call was like "Hello, there was a call eariler about a slow connection?" And at this point you have someone on the line who is interested in helping you, has power in the organisation to really fix things (because they're management or a senior tech) and they want to get your issue fixed to they don't lose your business. And THIS is when you really try to explain what's going on.
This was my experience. Perhaps it will work for you.
The question is about blocking port 80 outbound not port 80 inbound.
Proxy servers, They might not be cacheing 8080 or other Proxy ports. Check http://tools.rosinstrument.com/proxy/
Bouncers - You set this program on an external server on a port thats not filtered. You just point your browser at this IP/port and your outside your filtered isp. Check www.freshmeat.net
SSH, tunnel or route from an external box.
Really, If you cant go through it, go around it, either with software or networking.
-
Well, if crime fighters fight crime and fire fighters fight fire, what do freedom fighters fight? They never mention that part to us, do they? - George Carlin
The submitter seems a little confused about how http proxies are required to work. The ISP's proxy seems to be working exactly according to the standard. Taking an http with an absolute uri and redirecting it to the server specified by the uri is a MUST according to RFC 2068 (the HTTP/1.1 standard). Moreover, using a different name resolution system then the server for your client and expecting it to work is a MUST NOT as it can lead to proxy looping.
Support wide use of IPSEC.
Encrypted payload will prevent broken routers from looking into it.
That requires an external box, like the shell account the original comment mentioned. If you have that, you could use some more advanced schemes like routing only the SYN-packets for port 80 through your external account. This way you wouldn't cause three times the traffic like you do with a proxy (your connection plus twice the external connection).
The poster mentioned that he used OpenNIC which is an alternative DNS root. It is proper HTTP, but a transparent proxy that does not "see" domains in this namespace effectively block you from viewing webpages under this domain.
His own box is properly configured to do OpenNIC lookups, but the HTTP request to the (proper) webserver gets intercepted. Now the proxy has to do the real HTTP request, but the proxy does not know about the alternative domains and probably returns a "Host not found" error.
I haven't heard of free proxy servers supporting one of the alternative NICs and I doubt the ISP will be interesting in subscribing to such a service. I guess the only solution will be to convince a friend to set up a proxy on a box someplace else.
Some alternative roots have their own "real" Internet domain which acts as a gateway domain, for instance name.space has http://name.space.xs2.net/ (regular hostname) which enables non-subscribers to view http://name.space/ (namespace only), making the domains available globally. If OpenNIC provides such a service, an alternative solution could be to run some proxy at home and let it rewrite OpenNIC urls into "regular" URLs.
( ^_^)/
Once again, we're shown what happens when someone who doesn't know what they're doing gets into the pilots seat. In the past, I've seen the situation complicated by management demanding that something be implemented NOW. This leads to a new technology being put in place as an improperly implemented solution. In the end, when you consider the amount ot support work required for it, it end up being cheaper to do it right, but more slowly. For some reason too few people realize this.
Certifications would certainly seem to alleviate the frequency of many of these occurances, but in practice, I have seen too many certified employees who really don't understand the basic ideas of what they're trying to do. Sure they have a piece of paper stating that they passed the test, and may have paid $20k for a 1 month course in their given certification, but without real experience with the technology, it's all worthless. Combine this with management that believes that technical staff is merely there for implementation and not design or recommendation and you have a cycle where poor decisions are implemented 'just to get by' and are depended upon from that time on because noone who knows what they're doing has the authority to veto stupid decisions.
What we really need are more certifications that concentrate on ability and broad based knowledge than a specific way of doing things for not only admins, but also the managers of those admins. An incompetent manager has no business having the authority to tell a network admin to implement a new technology on a specific schedule. I fear that competent admins will soon become only slightly more respected than the guy who unclogs the toilet.
(1) Line up a serious alternative ISP. Talk to their sales department; see if they do the same thing.
(2) Talk to your ISP's sales department. Tell them your problem. Tell them you're ready to move. (Perhaps ask what the hit rate of the cache is, that is, if the overhead is worth it for them.) See if they offer any accomodation.
(3) Go with the ISP that does what you want.
If you're using them for DSL, you may not have a lot of choice.
(As others suggested, if host resolution is your issue, you could run a local proxy on your 127.0.0.1 interface that converts host names into addresses.)
Stupid job ads, weird spam, occasional insight at
How could a number outside 16 bits make it to a router since TCP only holds 16 bits for ports? If you wrap around to 80, you have 80, not 65616.
-Kevin
If you look at it from your ISP's standpoint transparent proxies aren't as evil as you make it sound.
99.9% of the ISPs clients aren't trying to do anything tricky, like this. Of those 99.9%, say, only 40% have a proxy server specified. These 40% get to enjoy faster web browsing--which is probably all they're doing anyway. The other 60% enjoy slightly less quick web browsing, but that's they're own fault, right? They're the only ones losing out, right?
Wrong. The ISP has to pay for bandwidth. The ISP doesn't like the proxy only because it makes browsing snappier, it likes the proxy because it also saves them on bandwidth costs! If the other 60% of the clients were using the proxy they might save 10%, or more, on total bandwidth costs.
You could think of it like this, too: that's 10% more bandwidth available for the clients at no additional cost to the company (apart from the capital for the proxy server). Yes, they're not perfect, but they make a difference. When you weigh the pros and cons, well, it's obviously going to be worth it for the ISPs to have it installed.
You could look around for an ISP that doesn't use a transparent proxy but, as you said, they're becoming more popular. Realise that they're not doing to squash your freedom, but instead to provide better service and to save money.
Here in Singapore, ISPs are required by law to block port 80, forcing all outgoing http requests to go through a proxy server (which filters out webpages which are deemed unsuitable for Singaporeans to view, including www.playboy.com), or to have a transparent proxy server blocking out such requests.
This has caused me many problems before, when my IP gets determined wrongly by the remote site (which naturally thinks takes the proxy server's IP for my IP address). Some applications don't like the transparent proxy either, for example Frontpage Extension (not my choice to use!), and an autopatching program which refused to download the latest version of a file, insisting on downloading only the file cached in the proxy server until the cache gets flushed.
The only real method of bypassing the proxy is to use another proxy server (since 8080 isn't blocked) outside the ISP's network. This tends to be really slow though.
I guess I have to live with this until the government one day realises that proxy servers cannot stop the people from viewing pr0n, and it's probably not worth maintaining the proxy servers to meet the demands of all the net users in Singapore, not to mention maintaining the list of sites to block.
Second, there's a lot of ways around it which involve tunnelling.
Tunnel to another box running a non-broken web cache. I used to tunnel my http traffic through ssh to my colocated boxes, which ran adzapper, and proxied through that.
Tunnel at the IP layer by running any IP-in-IP encapsulation. If you have some version of windows, for example, you might convince someone with a server to run a PPTP server for you somewhere and you could tunnel through that. There are even Free PPTP Servers for Linux available to help.
Find someone who runs a little proxier for their own net with socks, and bounce off their socks proxy. Someone you know no another ISP probably has Wingate or the like running, and if they allowed it (and on some older version, it will permit this by default), you could set your browsers SOCKS settings to bounce off their proxy server, and since SOCKS isn't on port 80, your ISP will probably ignore it.
There are also a number of things you might discuss with your ISP to resolve the issue.
Suggest that they switch to a less broken cache server. (Squid, anyone?)
Suggest that they exempt you specifically from the cache server by telling it to ignore your ip address.
Note that they have an obligation to make sure their caching software doesn't interfere with your browsing; so it will be necessary (and not cost-effective for them) for you to call for every problem you notice.
Obviously, you'll need to probably speak to a whole number of supervisors, and probably eventually get transferred to a "real engineer", and they will probably hack in a fix (like exempting you only) rather than truly deal with the problem.
If all else fails, then you may want to try issuing ultimatums, like, "If you can't fix this problem, then you can cancel my service." Tech support people are lazy, however, in some cases, and may just opt to cancel you. This is a harsh reality in the world of consumer bandwidth -- and it will be worse, soon, with bells closing their DSL lines to competition, meaning unless someone else builds a telephony infrastructure to you, you'll probably pick Cable vs 1 DSL provider, and if you don't like something at either of them, you're just out of luck.
The BEST solution that unfortunately will never be implemented is to allow specifying a port number in a DNS lookup. Then when the browser or e-mail looks up the address, one could also specify a port that you want.
Unfortunately, this ain't gonna happen without a rewrite of everything.
Sometimes it's best to just let stupid people be stupid.
doesn't allow ANY servers of ANY type
You'd better make sure that you always use PASV mode FTP then or you'd be breaking the rules!
Of course, the problem with transparent servers is when they're not, and your ISP seems to have one that isn't. Is it possible to find out what kind it is, either by telnetting to the thing and looking at headers or by asking the ISP, and can you do bug reports to the vendor to get them to fix their product?
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
My college has a similar set up because it saves an incredible amount of bandwidth. It's not to be mean, or malicious, or spy on your browsing habits, it's just to save bandwidth. And it does (I wish I had numbers to back this up, but I don't run the proxy).
There have been problems with the proxy in the past (it not returning any data) and there are still some minor issues, but on the whole it works well (in that you don't ever notice it).
It sounds like the ISP in question has a bug in their web cache code. If the web cache doesn't have the particular URL cached, it forward the request to the intended destination. I'd bet it's trying, but it can't lookup whatever OpenNIC URL is being specified (because it doesn't use OpenNIC). The ISP really should report this bug to the manufacturer.
My advice is this -- get the ISP on your side to fix the problem. They won't remove the proxy, and they shouldn't have to if the bug is fixed.
"Save the whales, feed the hungry, free the mallocs" -- author unknown
If you're using an OpenNIC DNS, shouldn't the client computer already have the IP address of the appropriate server? The request should be going out to an IP address, so I don't see why the ISP's proxy would want to interfere.
For example, if I typed http://64.28.67.150 into my browser, would the ISP's cache try to resolve that? or would it just forward the request to that address?
I have Cox cable modem (since @Home is dead). One day I found that I could not get to
http://www.cryptome.org via a web browser. I thought maybe the site was down, but I was able to get to it via a web proxy, and also ping www.cryptome.org was working. Does anyone know what this means? Just a bad DNS server along the way, or malicious blocking? Th web browser reports: HTTP Error 403 - Forbidden.
Again, before his packets get ANYWHERE, they have to go through that proxy his ISP is running. Setting another proxy would just be sending him through two.
"Upon attaching the waterblock to my penis, I began to notice that I know nothing about computers." -- JRockway
Qwest Communications, before selling dial-up and DSL customers to MSN, once had a transparent web proxy set up in Des Moines, Iowa. All outgoing HTTP traffic to port 80 was routed through the proxy.
The worst part was that when the proxy went down, packets continued to be routed to it, but tier 1 tech support personnel (located in another state--probably Minnesota) had no idea that the proxy even existed. The only way to work around it was to use a web proxy somewhere on the Internet that did not operate on port 80!
Qwest finally removed the transparent proxy shortly before switching customers to MSN. I eventually switched to Mediacom cable modem at home and McLeodUSA DSL at work.
AOL's transparent proxy is a little worse. It ignores the port and proxies anything that looks like HTTP. Of course, they deny having a transparent proxy, but I was able to watch packets leaving our network headed for AOL and then watch altered packets come back from AOL.
I stumbled across this when their proxy had some trouble with the cookies we were using and suddenly no one on AOL could use our service. A few minutes later they could again. Then they could not. During this time, I was running a packet logger on the outgoing traffic from our server and on the incoming traffic to a workstation I had connect to AOL. Everything worked find until the server sent the cookie. Then AOL suddenly stopped sending more packets. This occured on every port I tried, even ports reserved for other services.
The web cache is exhibiting correct behavior. When a forward proxy cache (transparent or not) gets a request in the form of GET http://www.site.com/ http/1.1, it will use the www.site.com address instead regardless of what original dns name you went to (www.google.com in your example). In the transparent case where the GET statement looks more like GET /content.html http/1.1, it will use the original destination address.
In other words, it's your client that's broken. See RFC 2616 for details.
The unfortunate truth is that more often than not, sites simply don't set their cache controls correctly. They forget that caches don't exist just on the server side but that they exist on the client side as well. Section 13 of RFC 2616 explains how they work in great detail and it really should be mandatory reading for any site administrator.
If you're still looking for more information on web caching, check out Content Delivery Networks by Scot Hull. It was just released and is available on Amazon. There is an enlightening section on web caching that should clearly explain why what you're seeing is in fact correct behavior.
First of all, this has nothing to do with "Port-80 routing", whatever that means. Second, if you ISP won't allow you to bypass their proxy, then your ISP is screwed, and any workaround you choose to implement will be so fragile and/or cumbersome as to be unusable. There is no third.
Pushin' 'n dealin', shovin' 'n stealin'
Idea: Have a local proxy insert an HTTP header line with a random value to make your payloads differ from other customers'.
My very lame ISP, AT&T Broadband, blocks my incoming port 80. What can I do to get around this?
Thanks!
If you want to find the IP address of a transparent proxy, simply point your web browser at a web page that will print out "your" IP address when you request a web page. Instead of printing the IP of your firewall or your host, it will print the transparent proxy's IP address.
For example:
After that, you may be able to do some more investigation into what kind of host it is and/or what kind of software it is running. (This is left as an excercise for the crac...err, reader.)
Normally what you do is to do layer 4 switching but note that you can do do switching on layer 7 as well, which means you can have the switch do url based switching so that a part of the url determines that it should get switched. This requires much more power and is mostly done for server switching like load balancing.
What happens in your case might be that they have placed a switch that can do at least layer 4 switching, between you and the internet.
What then is done is that all port 80 requests coming from the clients side(you) are re-directed to the proxy which means that http requests on other ports will not be cached. Note that anonymous ftp can also be proxied.
A "clever" proxy/switch solution can do ip-spoofing so the webserver gets your IP adr. and sends it back to you directly, but as there is a switch inbetween, it redirects the result to the proxy which then sends the result back to you.
A way to avoid it is to get a gateway somewhere that can channel your http traffic, you could set your browser to use this gateway as a proxy on any port. The switch will most likely not act on the traffic coming on this port an pass it though.
The easy way would be installing a proxy server on a box that you have access to on the outside and configure it so that it won't cache anything.
OK, this is a bit OT, but since you're from Singapore, I'm curious about something. I know that when filtering was proposed there, many people weren't happy about it. Has there ever been a move to form something akin to the EFF to protest this, or is the political situation still such that doing this would get you hauled into court by the government?
The whole political situation there baffles me. More repressive governments have been forced to reform by popular protests. Why hasn't it happened in Singapore? You'd think that, with the extent to which the country is connected to the rest of the world, people would see what's happened in places like Indonesia, Thailand, Yugoslavia, etc. and want to do the same.
That light you see at the end of the tunnel might be from an oncoming train.
No, most proxies listen on ports 8000 or 8080. Very few listen on port 80 and you don't need to use these. The transparent proxy decides that it wants to intercept the connection by looking at the destination port.
I just implemented a Cisco CE507 at work. WCCP on our core Cisco router redirects port 80 to the cache engine.
Save bandwidth and speeds deliverly of often viewed pages.
But ports other then 80 arent messed with at his isp, so using someone elses proxy not on port 80 would help
Hello,
How can you detect transparent proxying? Or opaque proxying?
Douglas Calvert
Run up their support costs until they start using
a non-broken proxy cache. Technical solutions are
nice, but they only fix the problem for *you*. If
you care about your peers, and the community of
users, solving the problem for *everyone* is much
to be preferred. Most users won't even understand
that they are being screwed by the ISP. They
depend on you to resolve the issue. Keep calling
support until they fix it.
-I like my women like I like my tea: green-
... are a pain in the rear. From time to time, the web proxy will just... die. No data from my box can go out on port 80 to any sites for a good 10-30 minutes. This is in addition to the usual crap with their gateways, which cause stalls in ALL data transfers at random intervals, for a solid 30-70 seconds. Ironically, that gateway problem stalls my large file downloads and makes it near impossible to view streaming media at any level of enjoyability... The two biggest features flaunted by broadband services like Comcast. Anyway, sorry for the OT rant. :P
Mozilla's a nice operating system, but it needs a better browser.
Due to lack of time I didn't get to implement that, but when I do, I'll try to first request the document from the ISP's proxy and only in case of an error to try the connect method.
If you can use a tunnel server, like IPSEC or PPTP or SSH, which lets you pick the IP address to send your IP packets to but doesn't interpret the packets itself, you'll mostly be ok (you'll still have to make sure to do your own DNS if you want to resolve on alternate roots.)
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Okay, find your own proxy to go through on some port besides 80.
Next, download one of the "Internet Radio" applications and find some good 160kbs channel in Poland. Connect to it and then keep it going 25x7. Depending on the size of your ISP that should more than nullify their savings by implementing a transparent proxy on your particular subnet.
I had MSN DSL for three years and had four computers connected anywhere from 160kbs to 128kbs 24x7 on streaming services after they overcharged me and took 12 months to refund my money. I'm sure they were glad when I left. Infrastructure costs are still roughly $.10 to $1.25 MB depending on the number of staff and quality of equipment. One user downloading several GB a month can really hurt.
It seems to me that ISPs use interception proxies to lower bandwidth costs. Here in Canada (Ontario at least), most of the big ISPs are talking about implementing bandwidth caps (5GB/month with excess charged at C$10/GB). I hope your ISP isn't doing both, as that would seem unnecessary and rather heavy-handed.
My previous ISP, Sympatico, used to have a transparent interception caching proxy. It was quite troublesome and more of a translucent crashing poxy server. I remember being unable to access starwars.com for two weeks once, even though everything else seemed fine. It was particularly annoying for people whose MTU was set too high (they needed 1454 or less) as they would constantly get timeouts on HTTP POST, such as when trying to send email from a web interface like Hotmail or Yahoo. It was also a constant source of problems for people trying to author their own personal web pages as it would cache them and not show their updates.
My current ISP, IStop.com, has an optional proxy. This is great! I normally use it, but if I have problems, I can switch to a direct connection. They run Squid and they also seem to have some sort of advert filter running. I get their logo (cached by my browser) or "This ad zapped" messages in place of at least 80% of web ads, which saves me lots of irritation, and both of us save lots of bandwidth. Incidentally, they also have reasonable bandwidth caps: 10GB non-local + 10GB local (mail, news, proxy, etc) per month, with excess charged at C$3/GB.
After a while, Sympatico reduced HTTP interception to large population centres like Toronto, Montreal and Ottawa. Finally, they stopped doing that too. I guess it was causing too many problems and costing them too much to deal with it. If my ISP were to introduce an interception proxy today, I would leave them immediately. It's just not worth the irritation and problems for the length of time it will take them to fix it or get rid of it. I do live in an area where there is plenty of DSL competition though.
So that would be my advice: switch ISPs immediately. Don't waste anymore time or effort on these guys.
You'd be hard pressed to break the rule, as some of these places actually block the incoming SYNs. Really.
Take texas.net DSL accounts circa 1999, for example.
I don't know if they still do. I purposely dumped them since they weren't willing to compromise on this point.
Considering one can get caned or even executed for fairly trivial things over there - I bet there hasn't been much protest. There also appear to be cultural factors at work - most people there would rather just "follow the rules", sort of like where the USA is heading. :(
Just because it CAN be done, doesn't mean it should!
There's nothing so wrong about the ISP intercepting port 80 connections BUT WHY DON'T THEY MAKE A NOTE OF THE ORIGINAL TARGET ADDRESS and use that as part of the caching key as well as the GET request?
AND then use the SAME IP when forwarding the connection.
blog.sam.liddicott.com
An ISP ("Internet Service Provider") offers me transport of IP packets between me and the rest of the internet. That's its primary job.
If the ISP doesn't do that, without interferance, then it's no ISP, because it didn't provide the serivce promised to you and you could take, in the extreme case, legal steps against them.
(For example, AOL is no ISP in my definition. It may offer HTTP and email services, but that doesn't make it an ISP.)
IANAL. Just my opinion, and I can only hope that judges would agree with my argumentation.
I am the network admin for a wireless isp that does transparent cacheing. If a user asks us to turn it off, we can disable it for their IP.
For more than 99% of our users, they don't know what routing or cacheing is, much less that it's happening. For those that actually have issues with the proxy it's a quick modification to our ipchains rules. So far we've only had 2 such requests. Also, we disable the cacheing for business class users by default.
I would hope that you would ask them to disable their transparent cacheing for you before doing something as rash as dropping them. It's my bet that most of their other users do not have this issue, and they may not even be aware that it is causing problems for you.
The original post describes the prediciment that she/he is in, but doesn't even say what is broken, exactly!
From the submission, it actually appears that the proxy is working exactly as configured. The end user, however, is breaking things himself by using nameservers other than his ISP's. That can't be described as a failure of the ISP by any means.
Proxy servers add a lot of value to any network larger than, say your 3l33t home rig. The two main purposes I use them for are to reduce overall bandwidth usage, and to insert some level of malware protection. I've saved myself, and my company a lot of headaches by blocking silly virus code requests.
It's nice that the post managed to include links to RFC, etc... it's too bad that they don't seem to really have an understanding of how networks, specifically the Internet, works.
As others have commented there are plenty of alternative ways to get around this like SSH tunnels, VPNs, third-party proxies, etc...
Just my own little $0.02 worth of a rant. Please drive through.
-buffy
Dr. Zowie's description of the problem sounded like something that can be worked around, at least for some cases - which is why he may need to work with the transparent-proxy's vendor and not just the ISP. The two big problems are using the correct DNS lookup, and having old data in caches. Cache aging strategy is a standard problem - some systems do better or worse jobs of managing it, and some give you workarounds. (For instance, the proxy I use at work seems to respond to "Reload" requests from the browser and refresh its contents.) DNS lookup problems are really a bug - if the browser sends an http request to 192.9.200.1 for foo.bar.altroot/zap.html, it's certainly easier to implement by having the proxy forget the original packet's IP address, see if they have the page cached, and re-resolve the URL if not, but it's also a bug - they could keep track of the IP address as well as the URL. The bugginess of the dumb approach is most apparent with alternate-roots, but it can also be a problem for URLs at domains with round-robin DNS, where requests can go to any of the IPs in the group, but multiple requests need to all go to the same server for consistency, either because of stateful requests or because the servers aren't running with identical content names (e.g. for dynamic pages.) (One can argue that the servers are buggy in that case, but that doesn't mean that the caching proxy's behaviour isn't also buggy.)
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
I think a light buld has just gone on above the kids head as to why his "work around" seems to work.
There are a few workarounds to the problem of devices that you do not wish to handle your traffic doing so.
I have seen tunneling via ip-ip, ssh, and other ipv4 protocols mentioned, however there is another option available, and that is to tunnel your traffic as ipv6 traffic over ipv4.
It does take a bit of time to set up, but if you can find an agreable ipv6 network provider to allow you to tunnel to their server, your traffic will not be handled by any transparent proxy server at your local ISP, regardless of the type of traffic that you are working with.
I am not sure how complete the ipv6 implementation for Windows is yet, or, depending upon which version of Windows you may be running, if it is even an option, but for users working with Linux and BSD, this should not be a significant issue.
Then again, I could be wrong.
-Rusty
You never know...
Gee, guess, what, genius? You start pretending that the DNS has a different root than other people do, and things break. Why is this surprising to you?
"Waaaaaaaaaa! I shot myself in the foot! All this blood, flesh and bone laying everywhere is clearly a CONSPIRACY to keep my OPEN SOURCE BULLET from violating the laws of physics! Someone alert Slashdot!"
My ISP (CTC) started doing this on my static dialup without warning. I noticed because 1) eBay pages suddenly required reloading in order to update (ie, if I quit the browser, and then went back to a dynamic eBay page, it was the same as before unless I reloaded the page). .. and then 2) I noticed when connecting to another machine, the address that showed up in the logs was not mine!
Anyway, after poking the machine I discovered it was a Cisco something or other. I also discovered that if you sent a malformed or invalid request, it would STOP transparent proxying for a few minutes!
So the solution I came up with was to telnet port 80 someplace (didn't matter where, because the proxy would pick it up) and type "PLEASE DON'T PROXY ME" and close the connection and then it would leave me alone for a few minutes.
Most of the time I left it on as the proxying seemed to speed up the usual day-to-day surfing. But you might want to try a script to do this automatically. Probably this is just an option the engineers forgot to turn off (I believe by law they must turn off all customer-friendly services :-).
After a few weeks of doing this, and making a few phone calls, the proxy mysteriously went away. Maybe they took my static dialup off the list, or they decided to do it for everybody. Whatever. I've been using Squid so it's pointless for me anyway.
Me and my roommate use Cox@Home in Phoenix, AZ and found that our port 80 traffic was blocked to our server here in the house. We contacted Cox and the rep confirmed that they are blocking all the traffic through that port since we shouldn't be running a server for HTTP (ie without purchasing Cox@Work). We get around it by referencing our ip to a service in an office where my friend is Sys Admin, and having him reroute the traffic to our server through another port.
We need a p2p network of Squid proxies.
We have a squid cache and during peak browsing, er, working times we see 40-50% cache hit rate.
I think the byte savings isn't quite as good as that, but I don't have any solid data to back that up.
The best I can say is that we had to shut the cache off for a day or so to do some maintenance and the help desk got a lot of calls about how "slow" the web was, in spite of the fact that not more than a few days prior we had *doubled* our internet bandwidth (single 1.5Mbit frame to MPP bonded dual pointtopoint).
I think that overall it provides much better bandwidth utilization (ie, fewer packets on the ISP link, even if the byte savings is only 10-20%) and the client browsing experience is a lot snappier.
Our ISP used to have a whole statewide squid cache hierarchy which you could tune your local squid cache into if you wanted to -- I wish they still did, the aggregated caching would have been very nice.
Not the removal, the separate availability. <mind mode=screensaver>You should be able to buy Linux and install it without Konqueror, and Konqueror without Linux. Oh, wait a minute... you can!</mind>
Just to rub the point home, you can buy and install Linux with or without graphics, with a different Graphics layer (such as Berlin), with a different window manager (such as FluxBox) and so on. All (modulo a few libraries) with or without Konqueror or one of a host of other browsers (Mozilla/Galeon/SkipStone, Netscape, Opera, Amaya, Mnemonic, OmniWeb etc).
Got time? Spend some of it coding or testing
Wrong. That's not what's happening. Ordinary proxying does use the modified GET request form where the URL is used in place of the URI. However, transparent proxying is different because the client is sending a URI, not a URL. And it's connecting to the origin server IP address directly, not to the proxy. The only way to identify the correct host is to use the IP address the client attempted to connect to. That's the transparent in "transparent proxy".
If a client does attempt to connect to some IP address, and a transparent proxy won't use that IP address because it thinks the origin server is at another address, that's wrong. But if it has no idea what the origin server IP address is at all, even though the client was indeed connecting to it, then that's doubly wrong. A message from the transparent proxy saying it cannot find the IP address is simply stupid because it has the IP address the client connected to, since this is a transparent proxy.
now we need to go OSS in diesel cars
If the user, despite ISP encouragement, chooses not to use a proxy, that should be his choice. He is paying for the bandwidth, and is assumed to be aware of the possible performance hit.
This was discussed in the vuln-dev mailing list after Comcast implemented transparent proxying.
This raised quite a stink when Comcast's logging habits were revealed. Oops.
There is obviously a performance degradation involved with re-resolving the address given to the cache server. Furthermore, requests now appear to be coming from the server, not the actual user -- potentially breaking host-based authentication systems.
I've also seen these cache systems horribly implemented. An IRC network that I administer recently starting checking for HTTP proxies on connection. This was performed by connecting to the remote user's host on certain ports (80, 3128, 8000, and 8080) and then issuing a CONNECT request. In more than one case, a blatantly stupid ISP redirected _incoming_ port 80 traffic to their server -- WITHOUT any sort of access restrictions on their proxy. Sort of ironic that they were probably using untold amounts of bandwidth for 1337 bounce kiddiots.
Proxying without consent is an Evil Thing.
Female Prison Rape in NY
Why not just use a port 8080 http proxy instead though? Most of our customers go through one without their knowledge, and those that know better simply turn this feature off. IE defaults to autodetecting an HTTP proxy anyhow, making for no configuration.
Jeremy
Port 80 is not the 'realm' of http. It's just commonly USED for http.
A transparent proxy *does* break standards. You are no longer buying an internet connection, you are buying a filtered, proxied, mutilated internet connection.
That aside, this is not the issue the guy is having.
He's trying to use an alternative DNS system.. but the proxy is using it's own.. so he is hostage to what his ISP wants to resolve things to.
As for standards.. the STANDARD is to route IP traffic, not analyze it, mangle it off to a transparent proxy, and then send it onwards.
I doubt it's 99.99999%. There are apparently quite a lot of people trading around on the .MP3 and the .DVD hidden domain networks. RIAA and MPAA people most likely have no idea how to get there, if they even know it exists. Do you?
now we need to go OSS in diesel cars
you could also setup a proxy on localhost that rewrites the Host header from 'Host: www.weird_ass.domain' to 'Host: www.weird_ass.domain.existing_domain.com', and then have the DNS server that resolves 'existing_domain.com' to reply with the IP for 'www.weird_ass.domain' when it gets a request for 'www.weird_ass.domain.existing_domain.com'. Maybe the maintainers of the 'weird_ass.domain' zone alredy have that.
You'll probably need a lot of custom code for something that can be fixed by changing ISPs tho.
--
Stay tuned for some shock and awe coming right up after this messages!
$OBSCURE_PORT_1 = obscure port # on your local machine
$OBSCURE_PORT_2 = obscure port # on machine outside firewall
On the machine where you have the shell account, download and compile the ucspi-tcp package, and micro_proxy. Put the tcpserver and micro_proxy binaries in your $PATH; throw everything else away.
To run the proxy:
From your local machine,
ssh -C -L $OBSCURE_PORT_1:127.0.0.1:$OBSCURE_PORT_2 -l [username] machine.where.you.have.shell.account.co.va
(or if you use some fancy Windoze SSH client, forward $OBSCURE_PORT_1 on your local machine to $OBSCURE_PORT_2 on the remote machine)
Once logged in, run tcpserver -DHlR 127.0.0.1 $OBSCURE_PORT_2 micro_proxy & on the remote machine
On your local machine, set your browser to use HTTP and HTTPS (IE)/SSL (Mozilla) proxies on host 127.0.0.1, port $OBSCURE_PORT_1
Surf to your heart's content.
Comment removed based on user account deletion
Comment removed based on user account deletion
How is that going to help ? He wants to use OpenNIC. I don't know of any free proxy servers that use OpenNIC which they'd have to.
That proxy machine or machines would make a great target for the script kiddies out there. Imagine if it were cracked open and reconfigured to return porn pictures for every tag proxied through it. Something like that would be such a great meta-hack.
When information is power, privacy is freedom.
This is don't with the Web Cache Communication Protocol (WCCP ) from his ROUTER. the command to find out if a Cisco router is WCCP enabled do a sh ip int (your int). Yo can look up the specs of the protocol to figure out how to try and bypass it. But you probably won't get ther by using another proxy(tried it), because you will still go through the original proxy configured at the router before going anywhere.
Saying Java is nice because it works on all OS's is like saying that anal sex is nice because it works on all genders.
When I pick up my phone and order a pizza, i don't expect the telco to redirect me to a "closer" pizza shop.
99% of ISP's customers have no idea what goes on when they click a link, so the ISP gets away with this bogus routing (proxying, whatever).
I'd like to see a good definition of what an "ISP" is. I have a feeling a lot of so called ISPs would fail the test.
Transparent? No way. There are two sides to a tcp connection, and the server side is completely blind to what's going on.
Call their tech support, waste their time til they figure out that you expect them to send your packets to their intended destination. It sucks that clueful helpdesk people get stuck in the middle of this, but it's the most effective thing a customer can do about it.
Proxy is required by law for end-users aka dial-ups and general public, but for business consumers its not madatory and the actual responsibility of internet access control is delegated down to the company itself. As for the general public, its up to them if they want to access pr0n, the policies ISPs imposes just follows the standard rules and guidelines from IDA. In short you as a general public defaults to no pr0n, but as a business user you can get yourself out of the tranparent proxy loop ;)
There's nothing to proetest about, would you go protesting to your ISP if you can't get to a pr0n site? Not very likely, unless you run that site. OT, Anything you do that doesn't coincides with the general public can get you into hot soup. So do you think that Singaporeans would choose to protest over filtering of internet access? Not very likely. The government only oppresses "trouble-makers", the few being oppressed are those whom have went out-of-line. The general public is very content with what the government is doing and has done for the last few decades and you can't compare it to those governments you mentioned.
The Singapore government is probably more concerned about stopping people accessing the numerous overseas sites run by the opposition movement. For those that don't follow Singapore politics it is one of those countries where the government brings specious lawsuits against opposition politians and elections are run in the manner of the old Soviet Union.
Of course since it is a capitalist pseudo-democracy this rarely gets comment in the western media. When it does the government has sued for libel under its mickey mouse libel laws in its kangeroo court system.
All phone calls made in Singapore are tapped and the government analyses the telephone call logs to see who is talking to whom. Its kinda the state that Ashcroft would like.
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/
Transparent proxying is a violation of IP routing, plain and simple. This has been discussed ad nauseum on the IETF WREC WG mailing list and the IETF main list.
Each (hostname,address) tuple can be a different site. The server needs to cache that way. I wouldn't consider doing it any other way. Maybe the proxy server you would write could be poisoned, but mine would not. Then the cache can be optimized by doing a DNS lookup (but not delaying the first request for this) to get the IP addresses. If there is more than one, the cache indentities for these would be merged and shared.
now we need to go OSS in diesel cars
WTF are you talking about? IPSEC and web proxies have been part of windows for years!
Important Stuff:
Please try to keep posts on topic.
Try to reply to other people comments instead of starting new threads.
Read other people's messages before posting your own to avoid simply duplicating what has already been said.
Use a clear subject that describes what your message is about.
Offtopic, Inflammatory, Inappropriate, Illegal, or Offensive comments might be moderated. (You can read everything, even moderated posts, by adjusting your threshold on the User Preferences Page)
A well written transparent proxy would forward the request to the original destination address of your packets if it suffered a cache miss. In that case, nothing would break. There is really no reason to perform a new lookup on the content of the host header. Of course, if a proxy behaved like that, it would be easy to force a transparent cache to store the wrong page for a particular url, by sending a request to the ip address of microsoft.com, with a host header of slashdot.org, for instance. A simple reverse lookup verification would work, so long as the action in the case of a failed lookup is to assume correctness, rather than failure. That would force all websites that cared to be set up with reverse dns correctly.
not that I am encouraging the use of such a technology, even if I did write one of them.
You think that's bad?
:-) Well, that's only if I choose not to homeschool 'em (why not? The only thing I experienced in school was learning to be a social idiot! Probably a lesson best not learned.)
I got an (approximately) 6 month suspension from the school's computer system because I used file manager! Yes, you know, the old explorer utility for windows 3.1... This was at EDSS in the WRDSB nee WCBE. There, as far back as 1996 they were filtering virtually everything. Even sites like Nintendo.com were filtered making one of the only fun ways to do an essay (on something you're interested in) boring. Funny enough, there were hints that they were going to filter FTP because it was "too much of a risk".
Fortunately my parents weren't broke, so between a small amount of my paper route and a lot of their help I was able to get a laptop.
I learned something from that experience, though. My kids will get their own laptops, and I will _refuse_ to allow them to use the school's computer systems. I'll even try it on as religious grounds if the school says they have to use theirs (maybe I can say I'm Amish?
If you could be told what you can see or read, then it follows that you could be told what to say or think - BoC
You could write a proxy (and optionally make it transparent) of your own that replaced the page name in HTTP requests with the text version of the IP address you were sending to. It's not a very elegant solution; going through two proxies to achieve the effect of going through none, but at least it would work. Alternatively, you could do what I would do in your situation, piss and moan to my ISP until they fixed it.
It isn't tunnelling or port forwarding or anything technical. This just wastes more bandwidth or cpu cycles.
Call your ISP, tell them that you want internet access, and explain what their proxy is preventing you from doing. Ask them how much it would cost for the service. If they are unwilling to supply the requested service, or charge to much for it, cancel your service and use another provider. Be sure to tell them why you changed.
Alot of small local ISPs were started because users wanted more features than AOL or the other providers. This is America, if you build a better product, customers will beat a path to your door to pay for it...until it is outlawed or you are put out of business by illegal (or recently legalized) tactics by the competition.
I've been getting Unable to acces webpages/DNS not found errors that appear to be due to Comcast. I keep hitting Refresh in the browser, and releasing and renewing the address assigned me.
This issue is manyfold. The reason the ISP is doing this is to improve the cache effeciency of many websites. If you resolve www.cnn.com, you will receive many answers back. If the proxy simply used the IP as a determiniation of what is 'unique' then it will end up copying each item from each IP, reducing the total effeciency of the cache. Instead, it will simply take the domain in either the request header, OR will make use of the host header, and then use that for further content retrieval. This in fact is fair. The issue that is being mentioned is when there is an alternate root server that you are attempting to retrieve content from. In this case, the ISP is broken (sorta) by not honoring the host names from this alternate DNS. There are several ways to solve this:
.com, .edu, .org, etc. This will allow the redirector to forward (without the proxy intervening) the request to the proper location, and the problem goes away. This actually would be very easy to do with a product such as the NetScaler RS6000 series boxes. Side advantages is that they can forward non-cacheable content directly, reducing the load on the proxies.
1) Get the ISP to use the Alternate root nameserver, allowing them to resolve the content (unlikely)
2) Attempt to contact the server without using a host header. In cases like this, the proxy will probably default to using the original source IP address that the connection contained in the first place, in order to handle programs that use port 80 for data retrieval but don't support host headers
3) Make use of an external proxy that doesn't do this interpretation (you probably won't find one)
4) Try to create a little proxy of some sort that you use that translates the host headers into an encoded format. This may get around the ISP's proxy, allowing you to use any DNS you want.
5) Suggest your ISP make use of a content aware redirecter that ensures that only the standard domains are sent to the proxy, such as
Transparent proxies are springing up everywhere, and you really can't avoid them much anymore. The issue you mentioned is actually new to me, but I can see where the problem can come in.
Yes, I've been told that Singaporeans are very "rules-oriented", but my understanding of the situation was that not only porn was subject to censorship, but also any political sites that the PAP sees as threatening its hold on power. Not to mention the other rules, such as required registration of locally-hosted political Web sites, which I'd imagine is a way to keep them offline. I can't imagine that everyone is happy with the situation, especially those who would like to stand in elections and change the government. Standing up and protesting for the right to see porn is one thing (although I'd say that what one does in one's home is their business, as long as no one else gets hurt), but stopping political speech is another thing entirely. And FWIW, at an ISP I worked at, we did get protests about filtering of porn sites. We didn't actually filter anything, but some users had filtering software on their computers without their knowledge (don't ask--they weren't the sharpest tools in the shed), and they were very clear in that they didn't want us interfereing with what they looked at. I also worked at an ISP that offered both unfiltered and filtered access, which I thought was the best solution. Whoever set up the account could tell us what kind of access they wanted, and only we could change it, so a user's kids couldn't figure out how to disable the filters. Well, I guess they could've used an outside proxy, but that wasn't our problem.
But I'm not so much making a statement here as asking the question: What would happen if one person or a group of people stated publicly that they thought that these filters are unfair and that they should be taken down, that they thought the rationale for their implementation was wrong, and that they do not serve the interests of ordinary citizens?
That light you see at the end of the tunnel might be from an oncoming train.
Desperate is right. The phone bill would eat him alive.
Yeah, I know, I used to use (in the PAST-TENSE) netcom/mindspring/earthlink, but know they have some ghey transparent web proxy cache that listens to EVERY port 80 request REGARDLESS of any actual webserver at any address. Doesn't this violate the DMCA? 1) They are "caching" (storing) other web-site's content thereby possibly violating copyright law. 2) You might have an old copy of the website. 3) It's slower if the page is not in their cache. 4) They can easily monitor what you are browsing and sell the info to advertisers. 5) They can change what you look at (Man-in-the-middle). Hmmm... maybe changing ISPs would be a better solution.
The biggest trick the devil pulled was letting lawyers become politicians so they can write the laws.
THIS IS JUST A BUNCH OF UNCONNECTED IDEAS THAT POPED UP WHILE ReADING THE LONG CHAIN OF POSTS FOR THIS THREADING two ideas for secure net. 1. sugar 2. secur lines "They hired a dozen level 1's and expect the level 3's to teach them how to use unix/etc/scripts. Its a fucking joke. You wonder why it takes forever to get tech support to fix something! The top tier of people staffed are mostly UNSKILLED... ..competent admins will soon become only slightly more respected than the guy who unclogs the toilet."
uhm... I don't get why would you get a human to teach a human how to use a computer.. why not get a human to teach a computer how to teach a human how to use a computer.. there is something called training software and the manual... also you can form buisness rings.. well what buisness doesn't and pool that one training file made in corel every couple weeks by one persona instead of having 10 or 15 tecs to instruction.. those tecs can then work on infrastucture building etc... playing pingpong whatever... really also that proxies idea.. having a online buffer between you know proxy server instead of proxy port having it work as an airlock simple stuff... only one end can be active at one time an you would just have to cut of encoded datasnatches not that I would have the slightest clue how they work .. I'm sure you could set up a software virtual airlock too some type of drive read partition partition wouldn't be difficult for all the smart people to make. or even two systems running on the same hardware one is virtual one is less virtual.. works in my mind. I already solved all the net hacking issues unfortunately I think people like it to be insecure... sorta. What I'm wondering is how the hell does everyone one know my join4free password?? for me when my connection doesnt work right I go through this checklist
1. shut down all viruses
2. disconnect and reconnect if it still doesn't really connect
3, decide how bad I want to use the net and if I really do restart other wise wait
4. if I still cant connect go through check list 4 times
minus step 4+
5. check various lines and hardware make sure I have connectivity with it
6. call my provider
7. problem get fixed other wise I add messed up interconnection to my list of things to get ultimate revenge for... uhm I mean just forget and take up water polo. Like isn't this thing suppose to hold up to nuclear winter and stuff I think the system should be able to handle a bunch of kids with windows xp :)a joke ahh(don't mind me its been a while)... What about an external modem connected to a cell phone and a oh nevermind....
(since @Home is dead). huh?
"My very lame ISP, AT&T Broadband, blocks my incoming port 80. What can I do to get around this?
Thanks!"
couldn't you just change your http:port? to port 81? or maybe there is a computer out there that acts a a genie type thingy you know like ask jeeves but nothing like it you know a server with software running that gets the files for you throught another protocol and sends them through another port theres a name for it I just cna't think of it right now anyway I think that the dutch should have atleast one.. .I think they have one of everything...... if people only listened hopefully they make a MAINN (multiuser artificial intellegence nueral network) for telephony response already its not like voice to text and text if processing doesn't exist... there's plently of motivated smart poeple out there what gives.
"RIAA and MPAA people most likely have no idea how to get there, if they even know it exists. Do you?" .. sure use sugar its all run by the riaa anyway isn't it since it is them that cause the underground trading.. just like the justice system makes murderers... but really what you do is trade a mp3 with someone who wants a mp3... hard concept eh? piracy doesn't make sense when there is more free music out there than people can listen to in a lifetime. It's funny you keep the artist that charge you for their music in buisness but people got to eat. It's gets messed up when you either know or are the people that are effected by piracy.. once again why capitalism sucks!!! bigtime .. one it creates lots of senseless laws.. oh I'll get off topic so I'll just stop there. hold on I can't be biased. nevermind I'm always going to be bi-as-ed.
"roughly $.10 to $1.25 MB depending on the number of staff and quality of equipment. One user downloading several GB a month can really hurt"
had no idea the cost I thought it was much less... hmm seems expensive for a ray of light. don't they have fish systems yet.. oh two.
"....never even heard of a real ISP that supported OpenNIC zones."
I don't like the name "OpenNIC" I'm trying to quit smoking and everytime there is any reference to damn...double damn(after reading buffy post) great anyway
everything will be mine one day anyway so what people build can only help me lead a happier life final judgment and understanding will reveal all. now I just wish everyone else thought this too.
this might be off topic too if a company is rich does that mean it is overcharging its customers?
I'm still wondering when people found out my join4free password
"You have effectively DOS'd the server, and the ISP has every right to consider your actions hostile"
they have rights to consider my actions hostile when I connect but in reality they shoudn't be worrying until the icbms from central tiawan are headed for their nodes. its not like they can't build the technologies right into the modems without a need for a server by isp or anything... all it would be is a latis of factory assigned serials each modem using .. oooh thats too naughty nevermind.,, wow I'm impressed I just reinvented the internet in my mind to make it even better .. I still think non represented police state of all people as police as the way to go. anyway
hmm sooner
If you just want it to work, and you don't want to switch ISPs right away, don't mess around with circumventing the damn thing. Just badger your ISP into adding the alternate root servers to the hints file on his DNS servers. This will give you a short term fix while you shop for a new ISP. Failing that, stop paying him and sighn his abuse, hostmaster, help@, etc. lists up for some free pr0n.
Incidentaly, I would like to hope this sort of thing is rare. Does anyone have any stats on how often this is done?
Get a webserver to tell you what it thinks is your IP, try www.whatismyipaddress.com
/NT/2k/XP or ifconfig on unix/linux
if it sees your ip as something different from your real external IP then your ISP is employing some sort of proxy.
You can find your ip by running WINIPCFG on win95/98/Me or ipconfig in
I see what you mean. You are sending traffic to a particular address based on your own DNS resolution, and if the traffic is proxied, you want it to be sent to your chosen destination, not that of the proxy.
In my opinion, the ISP is exhibiting correct behaviour.
Picture this: the object of the exercise with the transparent proxy is to cache pages and increase speed for the customer, right? I think it's already been agreed earlier in the thread that this is not entirely evil.
Let's say the proxy honours the destination IP address that you chose (I'm not sure how this would work in practice, but I'll go with it for now). It returns the web page from the server that your DNS picked, and caches it for the next guy.
Another customer requests a page with the same name. What if they're using a DNS root where the answer conflicts with yours? The customer gets the "wrong" web page. Because cached objects eventually expire, this means that the customer might get a completely different site dependent only on the time and date they happened request it.
The ISP doesn't use the same DNS root you do, so they can't begin to troubleshoot the problem.
I concede that the popular "alternate" DNS roots have few enough conflicts with the IANA-assigned roots at the minute, but even that is an irrelevancy - any solution that allows a customer to choose destination IP address on behalf of other customers opens up the ISP to a denial of service attack by a user less trustworthy than you or I. One could set up an arbitrary "root" server that resolves www.yahoo.com to my own site. Or google. Or some site that accepts credit card orders.
I can't see any scalable way out of this without the ISP picking one root, and sticking with it. If that is so, then I think this is a fundamental problem with split roots and, if you really want to use them, be fully aware of what you're getting yourself into. Turning off the transparent proxy will help this time, but you won't be able to rely on being able to talk to any server on the internet that doesn't use the same root as yours, even the servers you don't (usually) need to know exist.
Regards,
Dave
I have tried the above method with my ISP because I know that they use a transparent proxy (Squid). By telnetting to www.google.com 80, not only did I get redirected to /. by typing
but by typing:
However if I omitted the 'Host' header and simply typed 'GET / HTTP/1.0' I was taken to Google.
The proxy should perform it's own lookup in the first example but not in the second. Isn't this what's causing the problem?
I signed up for a
Purdue University does the same thing. They implemented a web cache server which was greatly underpowered. So when 6000 people attempted to go to www.yahoo.com the server crashed and IE faithfully put up numerous '404' errors for people. Anyways...don't feel like you are alone in this attempt by ISPs to reduce the amount of bandwidth they use.
It's usually cached by way of using an MD5 hash of the URL requested as an index in most caching servers (I know, I USED to work with a CDN that used several different tricks and we checked out loads of caching engines shortly before they shut it down, looking for an alternative to Squid). If you use the resolved IP address to place the request and use the HTTP header info only for caching index, you won't get a poisoned cache as you described it because IP address that you got the content from doesn't matter, only the request URL that got you there does.
/. discussion) in the process.
Now, as to why the "transparent" caches don't work like they should... Anyone that knows something of how they're set up would be able to tell you that there is no easy way to achieve the functionality to get the "correct" way with the typical setup. The typical setup usually involves router tricks to NAT the request such that it looks like a seperate caching server is the webserver for your request and then the caching server places the request accordingly.
Unfortunately, with dynamic content out there, a LOT of pages can't be directly cached (and there's nothing to make them so unless you do like epicRealm attempted to do with a CDN or what they and others are attempting to sell right now with an "app accelerator"- there's no current good protocol to tip the cache off that the content is stale so the content providers flag it as uncacheable...) so a "transparent" proxy is of some limited usefulness- unless you've got more than a couple of people placing requests for the same cacheable content, it inserts this big, fat latency and breaks a lot of things (like the subject of this
Unless it's truely transparent, being part of the router itself, it's probably more of a nuisance than a help, no matter what the ISP says to the contrary. I'd be finding a new ISP because they're being a little more than clueless.
I am not merely a "consumer" or a "taxpayer". I am a Citizen of the State of Texas
...most of the "transparent" proxies for HTTP tend to be router NAT hacks for a seperate caching server that is set up as a typical caching proxy. Since it's really a typical proxy with router tricks, it's operating in the usual proxy mode which then expects the proxy to do all DNS, etc. for the request, not the client.
It would be really contorted to achieve the "right" way, so nobody's bothered to come up with a caching engine that worked in the manner needed to do it truely transparently (Sitting on the router, etc.)
I am not merely a "consumer" or a "taxpayer". I am a Citizen of the State of Texas
Unless they're a large ISP, the only thing the cache is going to buy them a benefit on places that everyone hits that has static content because typically, most caches don't work well with the HTTP 1.1 cache hinting and it's difficult to set up for the HTTP 1.1 cache hinting so they usually send the dynamic content with pragma: no-cache in the headers and set the expires value to expire it immediately from the cache as stale. A cache is a web decellerator and buys NOTHING in the way bandwidth savings like most people think it does.
Amazon, Yahoo, et al. all set pragma: no-cache in the headers for a return request.
And you didn't pay attention, no less: His problem is he's using a different root DNS server than the so-called transparent proxy. Because of this, his browser will resolve and place requests correctly, but because the router is set up to NAT those requests and flip them to the caching proxy server, the request is then re-resolved for a DNS entry, etc. If they're not using the same DNS root, the whole thing breaks down.
I am not merely a "consumer" or a "taxpayer". I am a Citizen of the State of Texas
The host server can flag content as non-cacheable and the cache, if it's properly HTTP 1.0 or 1.1 compliant will merely relay the page to the requestor without caching it.
I am not merely a "consumer" or a "taxpayer". I am a Citizen of the State of Texas
Most proxies blindly resolve requests on behalf of the requestor because it's not really designed to be a "transparent" proxy- it's a router hack that makes it purportedly transparent. They are designed with the HTTP 1.0 or 1.1 proxy server specification from their respective RFC documents. Because of this, there is a relationship that is specified (i.e. the client browser places all HTTP requests to the cache, which then places the request as if it were the client browser. The client browser doesn't do DNS, etc. in this mode of operation) that is not present and is not assumed to be there with the un-proxied mode of operation.
This IS non-compliant with the RFC- it just "works" when you're using the same DNS server.
I am not merely a "consumer" or a "taxpayer". I am a Citizen of the State of Texas
IPSEC relies on unencrypted headers to work. This "transparent" proxy is a router hack that re-routes port 80 traffic for everything except the proxy server to the same. IPSEC would get flipped to the proxy and break down since it's not in the IPSEC session.
I am not merely a "consumer" or a "taxpayer". I am a Citizen of the State of Texas
If your client browsers are hitting static content sites, then YES, it's VERY effective. If your client browsers are hitting dynamic content sites, it's nowhere near as effective because the playground there is evil and broken. There's not a lot of fully HTTP 1.1 compliant caches out there (a requirement for a server to hint at expiry- needed for dynamic content...) and it's purely evil to set up the hinting for caches to work as intended- so nearly every dynamic content site out there (And that's the majority of the sites the average populace hits) set pragma: no-cache on the headers as well as setting an immediate expire time on the content. Dynamic content sites with average caching engines actually cause a degredation in browsing experience for the users as the caching engine never caches the dynamic content.
I am not merely a "consumer" or a "taxpayer". I am a Citizen of the State of Texas
It's a squid or similar server that is distinctly seperate from the router itself (A router COULD transparently proxy by being an interception proxy- but that's a lot more complicated and I don't think there's a lot of them about because they tend to be more expensive for some reason...).
I am not merely a "consumer" or a "taxpayer". I am a Citizen of the State of Texas
If the content is largely dynamic in nature, it won't get cached as the content providers tend to set pragma: no-cache on the headers and set the expires time in the past to force expiry to ensure fresh content. In the case of a LOT of stuff from Yahoo, Amazon, etc. you're going to find that a couple of pictures may cache, but the rest of the site will not be there for that reason.
I am not merely a "consumer" or a "taxpayer". I am a Citizen of the State of Texas
Largely speaking, dynamic content (i.e. app-server driven websites, like /., Amazon, etc.) don't get cached in servers because there's no clean, easy way to hint expiry as the content goes stale at unpredictable times. Because of this, these content providers tend to set pragma: no-cache in the header and set the expiry time to something in the past to force expiry from the cache as soon as it's served to the requesting client browser.
If you have your browsers all hitting static sites and content, it works very well. It's not so hot to miserable as they hit more dynamic content sites because of what I pointed about above.
I am not merely a "consumer" or a "taxpayer". I am a Citizen of the State of Texas
My former ISP cached on specific URLs. So, the most popular exact URLs were the ones being updated the most often, and those that didn't get much traffic usually lagged by up to two weeks in updates. The way I found to get around this was by tagging on a query string to whatever URL I was trying to access.
3 6&mode=thread&tid=95&123
would easily break the cache, and I could guarantee that I was seeing the most up-to-date information. I always thought it'd be nice to have this type of URL re-writing feature incorporated into a Squid.
This is obviously only a short term solution. I wouldn't want to have to modify URLs for the rest of my life.
Something like http://www.slashdot.org/?123 or http://slashdot.org/article.pl?sid=02/03/22/19482
So, if it just happens to allow mean, malicious, spying or filtering, "transparently" so that you can't get around it, THAT'S OK? Nope.
I don't care how easy it is to get around, it sucks. You could save even more bandwith by blocking debian mirrors, red hat's ftp sites and all manner of stuff that only affects a few of your users. No problem eh? Think of how much faster all the comercial crap will load up for all your "consumers". No thanks.
Friends don't help friends install M$ junk.
I used to run a proxy server for a major company in Australia, with approx 3,000 users on one system and 1,500 on another.
The 3000 user system were the Administrative departments, of a client, and the 1500 were the outsourcing organisation (the primary reason for the split).
Each system had approx 25Gb of cache storage, and both systems used user authentication. The purpose of authentication was necessary so that Individual Internet access could be revoked by managers, and the system was configured so that the passwords expired every month.
Also both systems were using proxy log scanning mechanisms, with information being reported back to the managers. However this was not reliable in picking up people who were visiting places where they should not, and the policing fell back on the managers.
However the system did provide some wonderful information about how well the proxy cache was being hit. For the Administrative department of 3000 users, the Cache systems only needed to completely download new 'objects' 50% of the time. This was calculated as a 25% bandwidth saving, after through investigation. This was not bad for a service that was consuming about 20Gb of downloaded data a month with the caching turned on. This was despite the large amount of dynamic content that existed at the time.
The 1500 techs were not getting as good figures, more like 25% hit rate, and 12.5% data saved. This was despite the size of the cache storage being the same, and the number of users being less.
No. ISP 'transparent' proxying works by a router notcing requests on a given port and silently redirecting to a proxy server.
The router will see port 80. It will redirect to the proxy. The proxy will do who-knows-what when confronted with an invalid URL. Hopefully, passthrough. Maybe. Maybe not.
[ BTW: You can do all this with iptables under Linux. Read the manpage. Try it out. Then you'll understand it. ]
Nonono. I've encountered this before myself (I was trying to use some other site's cookie on my site (don't ask ;) )- it works without the broken cache in between.).
/
/
The broken cache does this:
Intercept packets with destination IP address 1.2.3.4 and port 80.
Looks inside packet.
Sees
GET
Host: www.google.com
(other headers snipped)
It then IGNORES the 1.2.3.4 address, looks up www.google.com for itself (if DNS not cached yet).
Say it is successful and finds 216.239.51.100.
So it connects to 216.239.51.100 port 80 and says
GET
host: www.google.com
---
Now if it isn't successful with the dns lookup, you're screwed - it either gives you an error page or disconnects you - no message.
I believe the correct thing for the cache to do is to use the 1.2.3.4 address both in the cache index and in the outbound connection.
Now the issue with using the IP address in caching is that many sites have multiple IP addresses for the same address and the cache will have to treat them as different sites. This means you need more resources on the cache and performance is lower. So I figure the cache manufacturers figured that performance over correct behaviour was an acceptable tradeoff.
After all they can argue with their customers that the correct behaviour for a transparent proxy cache used to lower bandwidth usage is to lower the bandwidth usage even if it breaks rare situations like this.
Cheerio,
Link.
Cause thats what they do. 1 phone number for the entire city, and based on your caller id information, your call is routed to the nearest Pizza Hut.
It's a cool setup, memorize 1 number for the entire city, and no matter where you are, make the call and you get the right place of business.
Executive ability is deciding quickly and getting someone else to do the work. --John G. Pollard
Need a Linux consultant in New Orleans?