Slashdot Mirror

So it's time all the big ISP's to start using transparent squids or whatever, like the small ones do! They should slowdown (or even prevent) the /. - google effect.
Is your ISP forcing you to use a cache????

I agree with you specifically for the reason of not falling behind. I especially liked the "Have laters" reference in regards to class warfare in the US.

My reason for posting though was to point to a perfect case in point to me (of course I cannot find right now). It was regarding the OSS Squid Proxy which still doesn't have mainline IPv6 support. Yeah, you can get patches for it, but the core developers don't know IPv6, don't have IPv6 connections, and don't (at present) see any need to learn or pursue making it work with IPv6. It'll happen eventually, there will be the need when IPv6 reaches critical mass, or one of the core developers will integrate the patches into the mainline code, but until that point, the project stands to risk being superceeded by another project that might integrate IPv6 sooner and just at the time of critical mass (which will happen in Asia and Europe way before we have the Teir 1 ISPs moving on it in the US).

The next 1-2 years are really going to be key for early adopters of IPv6, IMHO. Even Cisco, who is touted to have had IPv6 support back in 2001 really didn't have it in mainline code for their non-ISP-class hugely-espensive routers until early 2003. Guess why they jumped in the ball? All these little start-up router companies in Japan were touting IPv6 support and Cisco was losing sales.

Did you know that even Windows XP doesn't have full-blown IPv6 support? Try to run it without an IPv4 DNS server, it won't work (and who can possibly remember an IPv6 address, heh). It lists the well-known DNS servers, but they don't work even if you have servers listening for those addresses (as is documented by Microsoft):
fec0:0:0:ffff::1
fec0:0:0:ffff::2
f ec0:0:0:ffff::3

Windows 2003 server is the first version to fully support an IPv6-only network. While we're not ready for that anytime soon, it just shows a short-sightedness, IMHO, on US vendors, and where I believe we're going to lose our edge if we keep it up.

Being pro-OSS, I don't mind it as much as Linux has had full-block IPv6 support since the 2.2 kernel days.

Get your ISP to install a transparent adzapping squid. Imagine how much faster the net will run if everybody did this: All of the content, none of the ads, and the most common web traffic handled mostly by local proxy hits.

The problem is everyone uses ISA, because no other firewall I have found can provide the following. 1. Basic Reporting on Users (jo used x MB and went to these web sites.) 2. Tie in to Active Directory, so we don't have to setup and maintain another directory.

You haven't looked very hard. My company uses squid, and it uses NTLM authentication against a windows 2000 domain. Users are authenticated automagically using the integrated IE authentication, and there's only one password store - the active directory on win2k server.

Squid logs everything. There are dozens of reporting tools (some free, some not) which can read squid log files and generate pretty graphs for management.

Squid has all sorts of detailed ACLs you can use to allow, disallow or redirect web browsing.

Squid is fast and free (aside from my time). How much did you pay for ISA?

Now, using ISA to manage non-web internet connections, that's something else entirely.

A popular solution is Privoxy's popup blocking chained with Squid's caching. In my opinion, that's the way to go. Privoxy by default also blocks ads and webbugs and nasty javascript and other things, but you can disable those features.

These could probably be configured as a transparent proxy if you don't want to set it up manually on users' computers, but speaking as a power user, I would never sign up with an ISP that stuck me with a proxy I couldn't avoid.

I'd never consider 802.11a at this point, the marketshare is all in 802.11b.

So, the next question is, should you go 802.11g (~54mbit), which is backward compatible with 802.11b?

How fast is your internet access going to be? Is it even going to be faster than 802.11b will provide (11mbit)? If users want to do laptop to laptop transfers, they should just use a crossover ethernet cable (100mbit). Hint: Most ADSL is 384kbit and will let you grab ~1mbit when things aren't busy at the ISP. 1mbit is "fast" for most folks.

IHMO, the owner should just see is as a way to increase his customer base for his existing revenue model, and have a cool thing to do when things are slow (but need to keep the other employees in check if things aren't getting done and he's not there all the time).

Futher, I'd suggest a caching engine like Squid, which can help with content filtering as well (say for employees, make them login before they can surf so you can track their time, etc.). Squidguard is my filter preference for filtering and there are many free content DBs online.

I'd be filtering porn sites, probably gambling, probably hate sites, etc., as I'd not want one customer offending another with graphic images. Of course, you could say MYOB and tell the guy to sit where no one can see his laptop, whatever...

NoCat is a good authentication model as well just so you can track folks in case something illegal is taking place.

I would recommend that you set clear rules with clear consequences if the rules are broken.

After that, I would recommend that all of your children access the internet via a single point. Setup a small home network if you do not already have one and then use a proxy server with a filter.

I run the internet filter at my work and we use Squid and Dansguardian. Dansguardian rules as a filter since it does true content filter. This will also help you out by logging every site with the user, ip accessed from.

Most of all, be fair, upfront, and consistent in your enforcement of the rules.

Download Squid and set it up to block all advert sites. I did and it makes the internet an altogether much more bearable experience.

You should be able to deal with a lot of your scalability issues by putting some kind of cache in front of your system, like Squid.

But it sounds like every page on your site is really dynamic. And thus uncachable.oy

But you want to replace it with a mostly static site, so obviously, not all that dynamic stuff is required.

Before you chuck the baby out with the bathwater:
- Can you revise your existing java site to serve most pages as essentially static?
- If so, will putting some cheap squid cache boxes in front of your main servers do the trick?

This technique really works, if you can do it.

Running Squid with a 256mb ram disk cache is all the speedup we need, and it does so without altering the data being fed from upstream.

Cache servers are a bad idea. The very idea is to try to be an end-all be-all to everyone who uses them. There are bug-fixes to some of the problem, but no way to solve the essential problem of the fact that MOST data on the web is dynamic now. Using cache servers with dynamic data is inviting difficulty and problem.

Cache servers are NOT a bad idea, they are a GREAT idea, and for this reason they are in wide use. I don't know what cache engine you were using, but it sure sounds like it sucked. Cache engines from Cisco and Network Appliance are said to be good. The open source squid proxy is an EXCELLENT cache engine from my experience, and I've yet to see any problems similar to what you've described. I've had squid in use at my kids school for the last year without any problems, and have been using it here at home for at least 3-4 years. It too can be configured to not download ads from known ad sources (like Doubleclick).
Using something like squid is very useful in a school environment especially. A teacher tells the class to all load a web page and it's only fetched from the web once, then all students are served it via the high speed cache. This saves both time and bandwidth.

A big flaw with windows update is that you have to get the whone 11mb per computer.

Put an HTTP proxy server between your LAN and the Internet. The first download will take a while, but your proxy should cache it so that subsequent downloads on other systems on your LAN will be much faster.

If you goto the squid page, you'll find the squid cache was started in 1995. Looks like prior art to me.

This is the earliest reference to transparent caching that I could find in the squid-users archive.

Oops.. that is, they filed in Sept. 1997, and I was using Squid before that.

Here's much of the early revision history of Squid.

Version 1.0beta1 was April 19, 1996, and that was based on Harvest which was even earlier.

Well, the first public release of squid 1.0.0 was if 1996 according to the Changelog>

That date appears to be prior to the patent application

Well, the first public release of squid 1.0.0 was if 1996 according to the Changelog>

That date appears to be prior to the patent application

I had thought of using Knoppix as an "appliance" distribution - I hadn't considered Apache, I was thinking for things like Squid, NTOP, Nagios, etc. Config information could be held on a floppy or flash drive.

Matt

Using Squid Proxy with squidGuard one can simply re-write MSN searchs and direct them to Google... Saves changing the default IE homepage and installing the google search bar as well:

rew srch-engines {
s@http://search.msn.com@http://www.google.com@ir
s@http://www.msn.com@http://news.google.com@iR
s@http://msn.com@http://news.google.com@iR
}

It's rather slick, if you ask me.

In a single company/school/lab/whatever, sharing small bandwidth costs across many servers & having everything be up-to-date would be great.

Squid does that.

Gator? Ohhh.. you mean one of those things my Privoxy and Squid combo block?

Why not just download and install Squid and enable authentication? Then just put that sucker out on your DMZ and you have authenticated web browsing to your local network.

This doesn't pertain to whether you should use DSL or Ethernet, but rather is a few things I've always thought ISPs should do. (I've had this almost life-long goal of starting an ISP for some reason...)

I own a domain, and use it primarily for the unlimited mail aliases. Every site I go to gets sitename@mydomain.com, which just forwards to my main address. If they start spamming, I can tell exactly who it is, and redirect (or block entirely) the mail. Why not give each customer a subdomain (customer.condo.com) where they get, say, 5 POP boxes, but unlimited aliases? Used effectively, this could *really* fight spam. (This is venturing more offtopic, but Cpanel seems to be the most popular web-based control panel; you could provide customers with some webspace and e-mail access. It's easy to use, but even great for geeks. You can get licenses for like $40/month, or possibly less.)

Another thing I've always thought ISPs should offer was NAT access. Rather than getting an external IP, they'd get an internal one and use your proxy. It'd save you from needing as many IPs, and it gives them great security -- unless you go out of your way to set it up, no one can connect to them. Of course you shouldn't force this upon people, but some people might *want* NAT. Offer it as a 'privacy' plan. (Heh, you could probably even charge extra, lol)

Something like Squid could really speed things up, especially if you only have a T1.

The last "If I ran an ISP..." item regards DNS. Maybe it's because Adelphia is so crappy (they have like 5 DNS servers, and whatever you have as primary ALWAYS goes down, so you're re-ordering the nameservers several times a week to make it work at all...), but I ended up using OpenNIC, which essentially is a 'democratic' TLD assigner; they have a lot of new TLDs not supported by 'real' DNS. (And, of course, lookups for regular TLDs work, too.) Not sure if you want to make it standard, but I'd be way impressed if an ISP gave me the choice of 'regular' DNS or OpenNIC DNS servers to use.

Oh! Don't forget to do your part and setup a good firewall. Another seemingly uncommon thing I've always thought ISPs should do was to do *good* egress filtering: filter traffic *leaving* your network too. I start to rant about this idea every time I read about a big DoS attack; if ISPs were more careful about what leaves their network, a lot of DoS attacks would simply get dropped at the attacker's ISP.

... is that Microsoft send out a good number of responses with a "Cache-Control: private" header. Any public cache storing these responses is in violation of RFC2616.

This posting from the squid-users mailing list sheds some more light on the issue.

If you were wanting to break the RFCs and were using squid, then you could probably modify src/http.c to return 1 for the relevant parts of the httpCachableReply function instead of 0, but that would be a "Bad Thing"(tm) when it came to RFC compliance.

I seem to remember trying this once:
use a redirector in squid and point customers to the local version on your website a la http://www.squid-cache.org/Doc/FAQ/FAQ-15.html Seems basic to me

That's great that you will give them money when you can, but we aren't talking about a donation thing here. We're talking about a product which gives you two options of use: buy it, or use the free version with ad banners. If you choose the free version and disable the ads, then you are stealing.

I'd like to echo many of the sentiments I've heard here already;

1. I disable ads in the other products I use; including running AdZapper with Squid to sanitize my browsing experience. So do many others.
2. Further to point #1, I don't make purchasing decisions based on advertising. Therefore, disabling ads for products I won't buy is stealing nothing from the advertiser, including the myth of "potential revenue".
3. Opera gives people the option of a "FREE Download". It's up to Opera to offer a limited version - which they have; it has screen real-estate taken up by the banner space. Whether there's a banner in that space or not is beside the point.

The notion of "stealing" from a company because you don't pay attention to their ads is mis-guided corporate brainwashing. A corporate television executive indicated that the viewers entered into an agreement with all TV networks to watch their commercials in return for watching their shows. That logic is just as flawed as distributing an otherwise fully functional product and hoping that people will decide to purchase it merely to rid themselves of the ads.

Long story short, the guy's using the version that he downloaded from the web. He is not using, nor is he advocating the use of the pirated version of the browser, so cut him a little slack huh?

Install Squid and the Squid ad-zapper plugin. Done and done. No more ads in Slashdot or other web sites.

Debian users are halfway there once they

apt-get install adzapper squid

If you have to use IIS for some reason, put a Squid proxy running on your favorite OS in front of it. It will save you a lot of trouble.

There's no NTLM authentication in Squid proxy either, and it makes no sense. I guarantee it would find much more use in the real world with NTLM.

Huh?

We have a Squid proxy server running right now using NTLM authentication with help from Winbind. The Squid FAQ has an entry here which explains how to implement it.

Hope this helps...

Why is it called Squid

.. if they could fit a squid in a fly.

• First of all, absolutely, positively install Squid. The benefits of using a proxy cache for your outbound HTTP traffic are too clear to not do it. More importantly, however, since all your HTTP is now being application-level proxied, the TCP packets must always, always, always be correct.
• Force everyone to use it. Sure, you could block port 80 on your IPmasq, but it's a better idea to transparently redirect all outbound port 80 traffic to Squid. There's a very good mini-HOWTO document on how to do this.
• If you want to be a real bastard (like me), tell Squid to rewrite the User-agent: HTTP headers, to make every web browser within your organization report as some web browser other than IE. This has the bonus effect of "reducing" IE's market share because all those web sites think your users are running Netscape on Linux (or whatever).

All you geek households need to go set up Squid and set it up to distribute the load. The Slashdot Effect can be eliminated within our lifetimes.

The Web was designed to work well with caching, particularly at organizational firewalls and peering points. It scales really well, and if you work inside a big company, or use a medium-sized ISP that has one, the first time somebody retrieves a given page, it's there for the next N users, and the bigger N is, the more chance that the first person got the page before Slashdot killed it. I've generally had much better success reading slashdotted sites from work, where I catch a cached version at the proxy, than from home. It requires a bit of computing horsepower at the firewall or gateway, but that's surprisingly cheap, and if bandwidth still costs you money, it can cut down significantly on costs when lots of people look at the same static content. It's obviously less useful for dynamic content, unless there's an easy way to tell if the dynamic content is the same for multiple viewers, but most web sites have content that's mostly static most of the time.

Akamai built a model that sells caching to content providers rather than viewers, which was technically interesting, and similar things have been done by their competitors such as AT&T, Digital Island, and Speedera, but if you're not doing a high-volume commercial site, and didn't expect to be slashdotted, it's the wrong model. Google's caching is fine, if Google catches it before Slashdot does and Slashdot actually points to it, but that's pretty rare. BitTorrent does a nice job of P2P caching and distribution of large files (its target application is things like CDs and big software distributions, and you'll find it used by some of the ETree Jam Band Music Download people - Bram's tested it for respectably-sized numbers of simultaneous downloaders (I think a few hundred, which is pretty big for CDs.)

If you look up "cache" in Google, the first entry you get is for Squid, which is also the first entry you get if you look up "squid".

All the annoying flash and shockwave ads are gone as well. Bliss!

apt-get install adzapper
And then set your proxy. TaDa!

It doesn't make very much sense to say "Should I use UltraMonkey or LVS?" as the latter is a piece of the former. There are other combinations of LVS+other stuff that you might put into that sentence: "Should I use Piranha or UltraMonkey?" or "Should I use UltraMonkey or Joe Macks LVS Config scripts?" or even "Should I build my own LVS scripts or use an existing framework?"

There are other HTTP load balancing options out there. Squid has a new branch in CVS called rproxy that handles multiple backend web servers very effectively with failure detection and other fun stuff (not to mention caching). Pound is a reverse proxy that does load balancing of HTTP traffic and SSL wrapping (most everything Squid can do for reverse proxying minus the caching features).

Balance is a generic TCP load balancer with some nice features. The best features being that it is simple and works on more platforms than just Linux and handles more than just the HTTP protocol. It probably has some disadvantages for some situations because it operates at a lower level than the HTTP proxies above, though it can probably do lots of the same things LVS does (I don't know very much about Balance).

Eddie is a neat framework written in Ericssons Erlang language. Seems to be dormant, but I think it is in pretty widespread use so is probably pretty stable.

Links:

LVS
Squid rproxy branch
Pound
Balance
Eddie

Here's why they are still up: Squid httpd-accelerator

Here's why they are still up: Squid httpd-accelerator

While we may disagree with the idea, as a person who is looking to become a parent in the next couple of years, this is a good idea. Have you seen kids on the Internet? They are very gullible, being brought up in the age of computers. They truely believe that they know all there is about computers and the adults have no idea how to use technology. They click on a ton of pop-ups, especially the ones that try to trick the user (You've got e-mail).

If you don't like it, don't go there, but let parents choose what forms of protection for their kids. (And here I am dreading the day I'm going to have to install Squid and SquidGuard on my home network).

So where are our open source alternatives to the proxy? Maybe the gaim and squid guys should get together? The only compelling feature I see is the direct-IM for internal users (i.e. not using up your internet bandwidth). The encryption might be nice, but you can already do that with gaim plugins. Everything else just seems like spyware, but maybe some companies need/want that?

Okay, now P2P to solve multi-cast routing of streaming live content like movies and audio broadcasts so if 50 people on a single ISP are watching a football game broadcast over the internet live efficiently that's cool. Web pages are trivial. ISP's, businesses, colleges, have all solved this problem for the end consumer. Shit, you can't go to www.yahoo.com anymore without hitting an Akaima server. All cable modem providers in my area use transparent squid proxies to speed up web browsing.

If P2P's big goal is to solve a trivial problem solved by the HTTP 1.1 spec, in conjunction with a couple of Open Source products, plus a couple of large business, I'd say P2P is about 3 years behind the times....

That said, P2P has some cool applications and will solve some cool problems, I don't think Web pages is one of them.

Kirby

here is a cache.

your two weeks of installation and configuration could have been avoided if you used debian and 'apt-get install squid' and spent 15 minutes reading the squid howto located here

people use microsoft because it is easy. people use linux because it works better (has a much higher price to performance ratio).

Just viewing the site launched endless popup ad windows some of which resized themselves to fill the whole screen, popped more windows when you closed the old ones, etc.

Really? I have cookies turned off, and popup protections on in Mozilla. To top it all off, I'm running adzap under my squid proxy, so I didn't see a single popup (or any of the many banner ads they intended me to see).

It's too bad, really. Some of them sounded kinda interesting... ;-)

Zeus really is great, it has some wonderful clustering features too, admin for the whole cluster can be done from one place. At the very least it's worth taking a look at the 30-day trial version to get an idea for how much work it would be to port the scripts across.

On a large site, you'll quite likely save the license cost by the decreased use of resources.

(AOLServer is a good server too, though it doesn't have the nice admin of Zeus there's a lot it can do and is also very efficient. I'm not sure whether it can throttle bandwidth by itself though).

You could look at using a combination of content acceleration and bandwidth pools in squid. I've used these features before and it actually works pretty well for static content. You can tune the caching params to allow for large files, etc.

Derek

But 'sticky' isn't 'zero affinity', is it? So what you really want it what the original poster suggested, a SSL-speaking proxy (eg Squid in SSL accellerator mode) that terminates the SSL session and forwards the request inside it to a cluster of non-SSL webservers (using RRDNS perhaps, or LVS if you want a 'smarter' solution). The downside there is your squid proxy is doing a lot of work, so you probably want to have a backup one and use something like heartbeat to fail-over to it if there's a problem with the first one.

I use Mozilla on all my boxen, but on my Windows box I have it saying its using the Linux version.

Nice little line to add to your user.js:
user_pref("general.useragent.override", "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.0) Gecko/20020529");

How's that for skewing web server statistics? :)

Those who have IE users on their network and want to skew results in favor of Mozilla can do so by using a web proxy (such as Squid). Hmmmm.. this could account for the large number of IE in web server logs. Many web proxies will send a set browser name to the web server, and I imagine most of these proxies are set to say its IE.

The best protection is install the antivirus in the server. Most antivirus apps can run in linux.

amavis
will call the antivirus from postfix ,sendmail and others.

viralator
scan the files downloaded with the web proxy squid

Both are worth install.