FHS and LSB actually seem to be well thought out to me. The thing I find interesting is that the one thing I most dislike, isn't really required... that being the SysV style init. The systems I run don't use it. They do have the directory structure in place, so if I did install a package that expects it, and wants to put a script there, it can. But it won't have any effect since I have a different init script system that's actually executed. One thing I like about this is that it leaves me in control of what starts at boot time.
I'm not going to complain to Redhat that they don't make their system exactly like Slackware. I know what their answer would be, and I use Slackware. So it seems to me that complaining to LSB that they don't do things exactly as I would is moot. I can do things my own way if I want, and they know I can... they know everyone can. It's still choice. The goal of FHS and LSB is to make available a particular well thought out choice that meets a variety of needs, and suitable for most people. If it were the case that what most people feel fine about using was required for everyone, we'd all be using Windows right now and FHS and LSB wouldn't even be an option.
It's the creation of the spec file that's a chore. I have to know what dependencies the package has to make it. If I know already, such as by RTFM's the original source package docs, then I know all I need to know to manage it without RPM. I still see making an RPM here as a redundant step.
I do some programming, but I still don't RPM-ify those programs... yet. But when someone comes up with an "autospec" "autorpm" program which figures out everything to make the RPM file so it becomes as trivial to make it as to install it, I might be more interested. Right now I'll still with "./configure" and "make install" which work just fine for me.
My sparc runs fine at run level 5. What would you like to see the various run levels be for? I changed mine around, and it looks like:
l0:0:wait:/etc/rc.sys 0 halt
l1:1:wait:/etc/rc.sys 1 single user mode
l2:2:wait:/etc/rc.sys 2 network admin mode
l3:3:wait:/etc/rc.sys 3 server mode
l4:4:wait:/etc/rc.sys 4 workstation mode console only
l5:5:wait:/etc/rc.sys 5 workstation mode X windows
l6:6:wait:/etc/rc.sys 6 reboot
Not some interactive self extracting tarball I can only use once unless I do the vendors job and package it myself (which unfortunately is necessary for modern sysadmins if they want to do their job properly).
If you already have the tarball, why are you trying to make a different package out of it? Don't forget that many applications are made for a variety of different systems. Linux isn't everything out there.
They use SysV init scripts which live in/etc/init.d. Again, I often have to do the vendors job for them and write the initscript myself. This sucks, I paid my money for a Linux app and I want a Linux app. This means you Sophos Mailmonitor.
I'll agree that it is annoying to have packages making assumptions about where they put boot scripts. But Linux is about choice. There is more than one standard to choose from. Sounds to me like you are trying to make cookie cutters.
Die symlinks, die. Linking correct locations to their incorrect locations should be as short term as possible. Yes, this means you Red Hat. Reverse the/etc/init.d ->/etc/rc.d/init.d symlinks now.
I hate symlinks, too. I want to get rid of all the symlinks in the whole/etc/rc.d. And guess what... I did!
After using many versions of Slackware, I finally tried Redhat at version 5.1. Actually I had tried it at a way earlier version and it never successfully installed. But 5.1 worked OK. The reason I tried it was I bought a Sun Sparc 5 and wanted to try Linux on it. Redhat seemed to be OK, so I later tried it on a couple other i386 systems, and that was working OK... for a while. As it turns out, I needed to make upgrades before RPMs became available (see next paragraph). I also needed to make some changes in how things were built. The RPM system started getting out of sync with what was actually installed. The system ran just fine, but soon it got to a point where some packages I was installing with RPM would not install because the RPM database thought things were not installed which actually were (but weren't installed from RPM, so I can understand why it didn't know this). So I ended up having to do forced installs. And that ended up making it more out of sync. By the time I had gotten to Redhat version 6.0, I was getting fed up with it. I switched back to Slackware (and Splack for Sun Sparc eventually came out and I use that, too) and am happy again, with well running systems. And I am now exploring LFS.
You say the system administrator should know how to package applications? Why the system administrator? I'd have thought you'd have expected the programmer to do that. If I get some package which is just a TGZ source file tree (because the developer was writing good portable code, but not using Linux to develop on), why should I, in the system administrator role, have to be one to make a package out of it? I'll agree it doesn't take more brains than needed to properly install the majority of source code, but I won't agree that it is easy (in terms of time spent) to do. At least I have the brains to actually check the requirements of what a given package I'm compiling needs, and make sure it is there by the time it is actually needed. The dependency may not be needed until it is run, so I have the flexibility of installing in whatever order I like. Also, some "dependencies" are option, and don't need to exist unless a feature is to be used that needs it. For example, if I'm not using LDAP for web site user logins, why would I need to make sure LDAP is installed if some module that would otherwise use it is smart enough to work right when I'm not using LDAP.
/opt is reserved for the installation of add-on application software packages.
A package to be installed in/opt must locate its static files in a separate/opt/<package> directory tree, where <package> is a name that describes the software package.
Doesn't look very depricated to me. I think the problem is your FHS link isn't really the FHS; it is the SAG (Systems Administrator Guide), which in section 4.1 clearly says it is loosely based on the FHS.
As for/usr/local, I do agree it should be off-limits to the distribution (besides setting it up if not already present). And packages in the package format of the distribution (e.g. RPM for Redhat, Mandrake, SuSE, etc... DEB for Debian and any like it... TGZ for Slackware... and so on) really should stay out of/usr/local. What/usr/local should be is whatever is local policy (FHS doesn't say it this way). Packages that the administrator really wants to be separate from the package management system, stuff compiled from source, stuff locally developed, all is eligible to be in/usr/local. My guess is the author of the article has no experience doing system administration combined with a decision making role where he might have to choose to do something slightly different than what everone else does.
It would cache the porn site, but that cache would be accessed only when someone else did the combination of connection to a www.microsoft.com address and gave a Host header request of www.somepornsite.com. Usually, the requests would have www.microsoft.com in the Host header, so a different point in the cache data would be accessed, and that point would have data gotten from some server operated by Microsoft. I don't think the reverse DNS lookup is useful here. The forward one for the given Host header domain name is good enough to get a set of IP addresses. If the IP address the client was accessing is different, then the data on this transfer is cached separate from the data gotten from any of those IP addresses from the forward DNS query (or maybe just not cached at all, at the choice of the admin). While your description seems like it might work, it would add DNS latency to the request latency. Keeping the cache separated by (address,hostname) seems to me to be the fastest solution.
Probably the biggest reason is that many people (seems most of them have shown up on/.) have difficulty understanding the caching logic to make it work right, or are dealing with some perverted intercept mechanism that loses the origin server (where the client is connecting to) IP address during translation. If the cache is not "in" the router, doing NAT to handoff the request to the cache on a separate box is wrong. But apparently some do this anyway.
The typical setup usually involves router tricks to NAT the request such that it looks like a seperate caching server is the webserver for your request...
If what you are describing is that the origin server IP address that the client browser was connecting to is lost by the time the caching proxy handles the request, then of course this is a problematic setup. This could be a big problem for HTTP/1.0 requests without the "Host" header, as at that point there is no way to figure out what the origin server is at all. Perhaps a caching proxy implementation will "cop out" and not support HTTP/1.0 at all. While I have not confirmed this, I do understand the Cisco WCCP protocol to intercept web requests and hand them off to a nearby cache server does include the IP address of the origin server. If this is so, and your network infrastructure is Cisco based, it might be prudent to use it, and demand that a cache vendor fully support it. I've seen it in squid, but I don't use it. I do have cache servers, but they are not doing intercepts. OTOH, they do use a DNS server with ORSC TLDs loaded into a local root zone file so I got to www.dev.null just fine even with the cache server being used.
Dude, this is bullshit. I'd like to see you write an efficient, scalable caching filesystem that allows for merging of one-way hashes (the URLs) after they've been stored (so, you plan to maintain a backward mapping? nice, efficient when you are doing thousands of URLs per second)
The hashes don't have to be changed. Once you have the set of IP addresses to merge, you simply pick one (the first one will do) and that becomes where all the data is stored. The cache entries for the other IPs will simply have an alias link to the first one. Those N-1 alias links will be set up at the time the collection of IP addresses is obtained. Although filesystem symlinks is an analogy to this, those would not be used. Instead a block of data would be used giving the canonical IP address to use for the cache, or give its already calculated hash. That data would also include the absolute expiration time based on the DNS TTL data, so the web site can change IPs.
At lookup time you have hash(IPaddress,hostname) and find the cache subtree for it. Instead of a subtree, you have gotten an alias object. Now you have hash(canonicalIPaddress,hostname) and lookup again, which now gives a real subtree. With hash(URI) you now can find the cached document. The difference is O(2) vs O(1) (whee). Along the way, of course, you check expiration, and schedule expired objects to be expunged.
Code is not needed because this is the analysis/design phase I'm describing. Good systems are done with some analysis and design (possibly going back to it in cycles as needed for extreme programming). Once you understand the design, you can code it. A serious cache server needs much much more analysis time, design time, coding time, and repeated cycles of development, than is possible while composing a/. response. What I described is necessarily brief because I have other projects to do which I am not dropping because one/. reader can't figure out the description of a design and has to see code to understand it. You don't have to take your words back because I don't care if you have them or not... they were worthless, so I put them in my Enron queue.
There certainly does seem to be confusion. I must say, the RFC authors made a poor choice of terms, although clearly it was a historical accident and this is probably the best that can be done to work around it. I've always used the term "transparent" to mean transparent in the sense that you can't tell it's there. I picked up on "interception" during this thread, but I had missed RFC3040 because my term greps must have missed it (concept indexing would be nice, but almost no one ever does that... it's quite hard to do... but maybe they have a different term for that, too). Ask 1000 people the meanings of some terms, and you'll get 1001 different answers.
Each (hostname,address) tuple can be a different site. The server needs to cache that way. I wouldn't consider doing it any other way. Maybe the proxy server you would write could be poisoned, but mine would not. Then the cache can be optimized by doing a DNS lookup (but not delaying the first request for this) to get the IP addresses. If there is more than one, the cache indentities for these would be merged and shared.
TCP/IP does not mandate the use of any specific root servers. This kind of thing doesn't even need to involve alternate DNS name spaces to have an impact. It's perfectly valid to connect to an IP address which is NOT listed in the A-records for a hostname which is also included on an HTTP "Host" header line. Yes, these are weird things, but they are valid internet protocol. If a business wants to assert that it is offering true internet service, then it needs to make it work exactly as if it were. If they can pull off doing a cache that ends up behaving to the end user exactly the same, great. But peakpeak.com didn't accomplish that. An intercepting transparent proxy server should always connect to the true origin server, and pass any "Host" headers through unchanged. It must use them to check for cache validly, along with the IP address. Even in the case of peakpeak.com's broken proxy, which was wanting to connect to whatever IP it got from DNS, should still fallback to the IP address the client was using if it gets nothing from DNS. It didn't even do that. That's very broken.
The proxy server, when operating in the role of an intercepting transparent proxy, should make the connection to the same IP address as the client did. And it should do it immediately if it has no cache entry for that (hostname,IPaddress) tuple. The proxy can then do a DNS lookup on that hostname later, to find out what IP addresses can have a merged/shared cache; later requests using one of those IP addresses would get the merged/shared cache, and those using other IP addresses would be cached separately.
So tell my why a proxy server can't do this.
Of course, it's not very likely the ISP will fix the problem very soon. Other solutions, such as an outside proxy, are likely to be the quicker workarounds. Ironically, this would defeat the bandwidth gains for the ISP, although for only an amount of that one customer. So the ISP shouldn't care as it's so small, right? If it turns out they do get angry that the customer is bypassing their cache proxy, then it's time for them to get their clue.
I doubt it's 99.99999%. There are apparently quite a lot of people trading around on the.MP3 and the.DVD hidden domain networks. RIAA and MPAA people most likely have no idea how to get there, if they even know it exists. Do you?
Even without WCCP, doing a straight route through a cache server box preserves the destination IP address in the IP header... it has to or packets cannot be delivered. You can also do a route-map based on destination port number and route only TCP packets destined for port 80 to one box and default everything else as usual. The destination IP is still preserved (but the ethernet MAC address is different as the routed-to IP address is used by ARP to find that MAC address). Now whether Squid can handle this case or not I haven't tested. But I do believe WCCP encapsulates the original destination IP address, so there's no reason not to correctly associate it with the request, and correctly connect to the intended origin server, and correctly distinquish cache data by IP address (which can be shared among those that DNS says are a common group with multiple A-records).
That's not necessary. In the case of a transparent proxy, which has intercepted the IP packet, the IP address of the origin server is available. It just has to be coded/configured to use it properly. In the case of the server running at peakpeak.com, this is not the case.
Also, this will not poison cache. You cache indexed by a tuple combining the hostname (if provided) and the IP address. The cache of several IPs can be combined in the case of DNS providing multiple IPs for the hostname. Thus if example.com resolves to 1.1.1.1 and 2.2.2.2 and 3.3.3.3 then all cache lookups that ask for example.com+1.1.1.1 or example.com+2.2.2.2 or example.com+3.3.3.3 can share the same cache subtree. But if some client asks for example.com in the HTTP "Host" header while connecting to 4.4.4.4, the cache lookup would be for example.com+4.4.4.4 which would be a separate cache subtree from the other 3.
Or the OP could look around for an ISP that does use transparent proxying correctly. This is not an unsolvable problem; all the proxy has to do is connect to the correct origin server, which is the IP address the client connected to. This is necessary because with transparent proxying, the GET request provides a URI, not a full URL.
Why do you think it cannot be worked around? Do you know what brand this proxy is? Do you work for peakpeak.com? Do you work for the vendor of that proxy? It sounds to me like someone set up a non-transparent proxy in a transparent role. But I would not rule out some vendor marketing department over-hyping some product as a solution for transparency when in fact it is good only for caching.
Wrong. That's not what's happening. Ordinary proxying does use the modified GET request form where the URL is used in place of the URI. However, transparent proxying is different because the client is sending a URI, not a URL. And it's connecting to the origin server IP address directly, not to the proxy. The only way to identify the correct host is to use the IP address the client attempted to connect to. That's the transparent in "transparent proxy".
If a client does attempt to connect to some IP address, and a transparent proxy won't use that IP address because it thinks the origin server is at another address, that's wrong. But if it has no idea what the origin server IP address is at all, even though the client was indeed connecting to it, then that's doubly wrong. A message from the transparent proxy saying it cannot find the IP address is simply stupid because it has the IP address the client connected to, since this is a transparent proxy.
A server running on a single IP address can serve multiple virtual hosts using the HTTP "Host" header to select among those configured. If you typed "http://64.28.67.150" then the client will send as one of the HTTP headers "Host: 64.28.67.150". The origin server (the one the ultimate has the requested document) uses that to select a virtual host, if so configured, or ignores it if not configured with virtual hosts. The proxy server, however, should respect the connecting IP address and use the resolved hostname only for cache matching purposes (so it can match a common cache among multiple IP addresses listed by DNS, if the destination IP is among them). So in the case of your question, using the IP address like that wouldn't make any difference. Where the difference lies in when you connect to a hostname in which your browser gets a different IP address than the proxy gets.
Proxies are not inherintly bad. As you point out, they do have important benefits. But a broken proxy, as peakpeak.com has, which fails to make the connection to the IP address the client tried to connect to, causes problems. The correct behaviour would be to do the DNS lookup to get a valid list of IP addresses. If the intended origin server the client tried to connect to is among those, then check the cache by hostname alone. If the IP addresses do not match, then check the cache for a combination of hostname and IP address. If the cache doesn't have the requested document, then connect to the proper IP address and fetch it.
I don't know which proxy servers (can) do this properly. Hopefully someone can post (a link to) a proxy server review with this much detail. I do agree the ISP needs to fix the proxy, and it shouldn't be necessary to remove the proxy to accomplish it. If they can't fix the proxy, then a workaround is to route the one customer around it (route-map in Cisco IOS can do this).
Slashdot Mirror
FHS and LSB actually seem to be well thought out to me. The thing I find interesting is that the one thing I most dislike, isn't really required ... that being the SysV style init. The systems I run don't use it. They do have the directory structure in place, so if I did install a package that expects it, and wants to put a script there, it can. But it won't have any effect since I have a different init script system that's actually executed. One thing I like about this is that it leaves me in control of what starts at boot time.
I'm not going to complain to Redhat that they don't make their system exactly like Slackware. I know what their answer would be, and I use Slackware. So it seems to me that complaining to LSB that they don't do things exactly as I would is moot. I can do things my own way if I want, and they know I can ... they know everyone can. It's still choice. The goal of FHS and LSB is to make available a particular well thought out choice that meets a variety of needs, and suitable for most people. If it were the case that what most people feel fine about using was required for everyone, we'd all be using Windows right now and FHS and LSB wouldn't even be an option.
It's the creation of the spec file that's a chore. I have to know what dependencies the package has to make it. If I know already, such as by RTFM's the original source package docs, then I know all I need to know to manage it without RPM. I still see making an RPM here as a redundant step.
I do some programming, but I still don't RPM-ify those programs ... yet. But when someone comes up with an "autospec" "autorpm" program which figures out everything to make the RPM file so it becomes as trivial to make it as to install it, I might be more interested. Right now I'll still with "./configure" and "make install" which work just fine for me.
Just send it to EFF and they will take care of it. Thank you for your support.
My sparc runs fine at run level 5. What would you like to see the various run levels be for? I changed mine around, and it looks like:
If you already have the tarball, why are you trying to make a different package out of it? Don't forget that many applications are made for a variety of different systems. Linux isn't everything out there.
I'll agree that it is annoying to have packages making assumptions about where they put boot scripts. But Linux is about choice. There is more than one standard to choose from. Sounds to me like you are trying to make cookie cutters.
I hate symlinks, too. I want to get rid of all the symlinks in the whole /etc/rc.d. And guess what ... I did!
After using many versions of Slackware, I finally tried Redhat at version 5.1. Actually I had tried it at a way earlier version and it never successfully installed. But 5.1 worked OK. The reason I tried it was I bought a Sun Sparc 5 and wanted to try Linux on it. Redhat seemed to be OK, so I later tried it on a couple other i386 systems, and that was working OK ... for a while. As it turns out, I needed to make upgrades before RPMs became available (see next paragraph). I also needed to make some changes in how things were built. The RPM system started getting out of sync with what was actually installed. The system ran just fine, but soon it got to a point where some packages I was installing with RPM would not install because the RPM database thought things were not installed which actually were (but weren't installed from RPM, so I can understand why it didn't know this). So I ended up having to do forced installs. And that ended up making it more out of sync. By the time I had gotten to Redhat version 6.0, I was getting fed up with it. I switched back to Slackware (and Splack for Sun Sparc eventually came out and I use that, too) and am happy again, with well running systems. And I am now exploring LFS.
You say the system administrator should know how to package applications? Why the system administrator? I'd have thought you'd have expected the programmer to do that. If I get some package which is just a TGZ source file tree (because the developer was writing good portable code, but not using Linux to develop on), why should I, in the system administrator role, have to be one to make a package out of it? I'll agree it doesn't take more brains than needed to properly install the majority of source code, but I won't agree that it is easy (in terms of time spent) to do. At least I have the brains to actually check the requirements of what a given package I'm compiling needs, and make sure it is there by the time it is actually needed. The dependency may not be needed until it is run, so I have the flexibility of installing in whatever order I like. Also, some "dependencies" are option, and don't need to exist unless a feature is to be used that needs it. For example, if I'm not using LDAP for web site user logins, why would I need to make sure LDAP is installed if some module that would otherwise use it is smart enough to work right when I'm not using LDAP.
/opt is in FHS 2.2 at secton 3.12. It begins:
Doesn't look very depricated to me. I think the problem is your FHS link isn't really the FHS; it is the SAG (Systems Administrator Guide), which in section 4.1 clearly says it is loosely based on the FHS.
As for /usr/local, I do agree it should be off-limits to the distribution (besides setting it up if not already present). And packages in the package format of the distribution (e.g. RPM for Redhat, Mandrake, SuSE, etc ... DEB for Debian and any like it ... TGZ for Slackware ... and so on) really should stay out of /usr/local. What /usr/local should be is whatever is local policy (FHS doesn't say it this way). Packages that the administrator really wants to be separate from the package management system, stuff compiled from source, stuff locally developed, all is eligible to be in /usr/local. My guess is the author of the article has no experience doing system administration combined with a decision making role where he might have to choose to do something slightly different than what everone else does.
Who told you that? ICANN?
It would cache the porn site, but that cache would be accessed only when someone else did the combination of connection to a www.microsoft.com address and gave a Host header request of www.somepornsite.com. Usually, the requests would have www.microsoft.com in the Host header, so a different point in the cache data would be accessed, and that point would have data gotten from some server operated by Microsoft. I don't think the reverse DNS lookup is useful here. The forward one for the given Host header domain name is good enough to get a set of IP addresses. If the IP address the client was accessing is different, then the data on this transfer is cached separate from the data gotten from any of those IP addresses from the forward DNS query (or maybe just not cached at all, at the choice of the admin). While your description seems like it might work, it would add DNS latency to the request latency. Keeping the cache separated by (address,hostname) seems to me to be the fastest solution.
Probably the biggest reason is that many people (seems most of them have shown up on /.) have difficulty understanding the caching logic to make it work right, or are dealing with some perverted intercept mechanism that loses the origin server (where the client is connecting to) IP address during translation. If the cache is not "in" the router, doing NAT to handoff the request to the cache on a separate box is wrong. But apparently some do this anyway.
If what you are describing is that the origin server IP address that the client browser was connecting to is lost by the time the caching proxy handles the request, then of course this is a problematic setup. This could be a big problem for HTTP/1.0 requests without the "Host" header, as at that point there is no way to figure out what the origin server is at all. Perhaps a caching proxy implementation will "cop out" and not support HTTP/1.0 at all. While I have not confirmed this, I do understand the Cisco WCCP protocol to intercept web requests and hand them off to a nearby cache server does include the IP address of the origin server. If this is so, and your network infrastructure is Cisco based, it might be prudent to use it, and demand that a cache vendor fully support it. I've seen it in squid, but I don't use it. I do have cache servers, but they are not doing intercepts. OTOH, they do use a DNS server with ORSC TLDs loaded into a local root zone file so I got to www.dev.null just fine even with the cache server being used.
The hashes don't have to be changed. Once you have the set of IP addresses to merge, you simply pick one (the first one will do) and that becomes where all the data is stored. The cache entries for the other IPs will simply have an alias link to the first one. Those N-1 alias links will be set up at the time the collection of IP addresses is obtained. Although filesystem symlinks is an analogy to this, those would not be used. Instead a block of data would be used giving the canonical IP address to use for the cache, or give its already calculated hash. That data would also include the absolute expiration time based on the DNS TTL data, so the web site can change IPs.
At lookup time you have hash(IPaddress,hostname) and find the cache subtree for it. Instead of a subtree, you have gotten an alias object. Now you have hash(canonicalIPaddress,hostname) and lookup again, which now gives a real subtree. With hash(URI) you now can find the cached document. The difference is O(2) vs O(1) (whee). Along the way, of course, you check expiration, and schedule expired objects to be expunged.
Code is not needed because this is the analysis/design phase I'm describing. Good systems are done with some analysis and design (possibly going back to it in cycles as needed for extreme programming). Once you understand the design, you can code it. A serious cache server needs much much more analysis time, design time, coding time, and repeated cycles of development, than is possible while composing a /. response. What I described is necessarily brief because I have other projects to do which I am not dropping because one /. reader can't figure out the description of a design and has to see code to understand it. You don't have to take your words back because I don't care if you have them or not ... they were worthless, so I put them in my Enron queue.
There certainly does seem to be confusion. I must say, the RFC authors made a poor choice of terms, although clearly it was a historical accident and this is probably the best that can be done to work around it. I've always used the term "transparent" to mean transparent in the sense that you can't tell it's there. I picked up on "interception" during this thread, but I had missed RFC3040 because my term greps must have missed it (concept indexing would be nice, but almost no one ever does that ... it's quite hard to do ... but maybe they have a different term for that, too). Ask 1000 people the meanings of some terms, and you'll get 1001 different answers.
Each (hostname,address) tuple can be a different site. The server needs to cache that way. I wouldn't consider doing it any other way. Maybe the proxy server you would write could be poisoned, but mine would not. Then the cache can be optimized by doing a DNS lookup (but not delaying the first request for this) to get the IP addresses. If there is more than one, the cache indentities for these would be merged and shared.
What makes you think it suspended proxying for all the other customers/user when just one customer/user does this?
TCP/IP does not mandate the use of any specific root servers. This kind of thing doesn't even need to involve alternate DNS name spaces to have an impact. It's perfectly valid to connect to an IP address which is NOT listed in the A-records for a hostname which is also included on an HTTP "Host" header line. Yes, these are weird things, but they are valid internet protocol. If a business wants to assert that it is offering true internet service, then it needs to make it work exactly as if it were. If they can pull off doing a cache that ends up behaving to the end user exactly the same, great. But peakpeak.com didn't accomplish that. An intercepting transparent proxy server should always connect to the true origin server, and pass any "Host" headers through unchanged. It must use them to check for cache validly, along with the IP address. Even in the case of peakpeak.com's broken proxy, which was wanting to connect to whatever IP it got from DNS, should still fallback to the IP address the client was using if it gets nothing from DNS. It didn't even do that. That's very broken.
The proxy server, when operating in the role of an intercepting transparent proxy, should make the connection to the same IP address as the client did. And it should do it immediately if it has no cache entry for that (hostname,IPaddress) tuple. The proxy can then do a DNS lookup on that hostname later, to find out what IP addresses can have a merged/shared cache; later requests using one of those IP addresses would get the merged/shared cache, and those using other IP addresses would be cached separately.
So tell my why a proxy server can't do this.
Of course, it's not very likely the ISP will fix the problem very soon. Other solutions, such as an outside proxy, are likely to be the quicker workarounds. Ironically, this would defeat the bandwidth gains for the ISP, although for only an amount of that one customer. So the ISP shouldn't care as it's so small, right? If it turns out they do get angry that the customer is bypassing their cache proxy, then it's time for them to get their clue.
I doubt it's 99.99999%. There are apparently quite a lot of people trading around on the .MP3 and the .DVD hidden domain networks. RIAA and MPAA people most likely have no idea how to get there, if they even know it exists. Do you?
Even without WCCP, doing a straight route through a cache server box preserves the destination IP address in the IP header ... it has to or packets cannot be delivered. You can also do a route-map based on destination port number and route only TCP packets destined for port 80 to one box and default everything else as usual. The destination IP is still preserved (but the ethernet MAC address is different as the routed-to IP address is used by ARP to find that MAC address). Now whether Squid can handle this case or not I haven't tested. But I do believe WCCP encapsulates the original destination IP address, so there's no reason not to correctly associate it with the request, and correctly connect to the intended origin server, and correctly distinquish cache data by IP address (which can be shared among those that DNS says are a common group with multiple A-records).
That's not necessary. In the case of a transparent proxy, which has intercepted the IP packet, the IP address of the origin server is available. It just has to be coded/configured to use it properly. In the case of the server running at peakpeak.com, this is not the case.
Also, this will not poison cache. You cache indexed by a tuple combining the hostname (if provided) and the IP address. The cache of several IPs can be combined in the case of DNS providing multiple IPs for the hostname. Thus if example.com resolves to 1.1.1.1 and 2.2.2.2 and 3.3.3.3 then all cache lookups that ask for example.com+1.1.1.1 or example.com+2.2.2.2 or example.com+3.3.3.3 can share the same cache subtree. But if some client asks for example.com in the HTTP "Host" header while connecting to 4.4.4.4, the cache lookup would be for example.com+4.4.4.4 which would be a separate cache subtree from the other 3.
Or the OP could look around for an ISP that does use transparent proxying correctly. This is not an unsolvable problem; all the proxy has to do is connect to the correct origin server, which is the IP address the client connected to. This is necessary because with transparent proxying, the GET request provides a URI, not a full URL.
Why do you think it cannot be worked around? Do you know what brand this proxy is? Do you work for peakpeak.com? Do you work for the vendor of that proxy? It sounds to me like someone set up a non-transparent proxy in a transparent role. But I would not rule out some vendor marketing department over-hyping some product as a solution for transparency when in fact it is good only for caching.
Wrong. That's not what's happening. Ordinary proxying does use the modified GET request form where the URL is used in place of the URI. However, transparent proxying is different because the client is sending a URI, not a URL. And it's connecting to the origin server IP address directly, not to the proxy. The only way to identify the correct host is to use the IP address the client attempted to connect to. That's the transparent in "transparent proxy".
If a client does attempt to connect to some IP address, and a transparent proxy won't use that IP address because it thinks the origin server is at another address, that's wrong. But if it has no idea what the origin server IP address is at all, even though the client was indeed connecting to it, then that's doubly wrong. A message from the transparent proxy saying it cannot find the IP address is simply stupid because it has the IP address the client connected to, since this is a transparent proxy.
A server running on a single IP address can serve multiple virtual hosts using the HTTP "Host" header to select among those configured. If you typed "http://64.28.67.150" then the client will send as one of the HTTP headers "Host: 64.28.67.150". The origin server (the one the ultimate has the requested document) uses that to select a virtual host, if so configured, or ignores it if not configured with virtual hosts. The proxy server, however, should respect the connecting IP address and use the resolved hostname only for cache matching purposes (so it can match a common cache among multiple IP addresses listed by DNS, if the destination IP is among them). So in the case of your question, using the IP address like that wouldn't make any difference. Where the difference lies in when you connect to a hostname in which your browser gets a different IP address than the proxy gets.
Proxies are not inherintly bad. As you point out, they do have important benefits. But a broken proxy, as peakpeak.com has, which fails to make the connection to the IP address the client tried to connect to, causes problems. The correct behaviour would be to do the DNS lookup to get a valid list of IP addresses. If the intended origin server the client tried to connect to is among those, then check the cache by hostname alone. If the IP addresses do not match, then check the cache for a combination of hostname and IP address. If the cache doesn't have the requested document, then connect to the proper IP address and fetch it.
I don't know which proxy servers (can) do this properly. Hopefully someone can post (a link to) a proxy server review with this much detail. I do agree the ISP needs to fix the proxy, and it shouldn't be necessary to remove the proxy to accomplish it. If they can't fix the proxy, then a workaround is to route the one customer around it (route-map in Cisco IOS can do this).