Slashdot Mirror

An important part of network design is understanding what traffic is going to "nearby" locations, and designing things so most traffic stays local and doesn't use expensive or scarce facilities - things like putting big hulking routers in San Francisco and San Jose so traffic between Silicon Valley companies stays in the South Bay and Multimedia Gulch companies stays in the City without needing to use too much bandwidth around the Bay, much less sending copies of all of it on three-part-carbon forms to NSA's Fort Meade, Ashcroft's J. Edgar Hoover building, and Dick Cheney's stockbroker before delivering it.

That doesn't mean that there weren't rumors from reputable sources a few years ago about active wiretaps on MAE-West sending extra copies of some packets to somebody else, or that the Russian renamed-KGB's 1998ish SORM (another URL) project didn't try to force Russian ISPs to build a full-sized wiretap feed to them (at the ISPs' expense, of course) or that there aren't Eurocrats trying to do the same thing in their countries today. And then there's the whole Echelon Wiretapping System. But it's still impractical for them to force ISPs to deliver everything everybody's reading or emailing, though I'll be happy to send them copies of most of my spam if they'd like.

On the other hand, the publicly-accessible parts of the web aren't all that big. The Wayback Machine has a copy of all of it, with reasonable samples going back a long time, and Google and the other search engines crawl it periodically, and AllTheWeb.com presumably claims to have All The Web.

Since it isn't using its own engine, you should be fine...

Yes it is. What many of us think of as the "HotBot engine" is actually the Inktomi engine, which is still available on HotBot and is in fact the default. (The others are FAST, Teoma, and the yardstick by which all others are measured.)

-Fantastic Lad

"Evil," says Google CEO Eric Schmidt, "is what Sergey says is evil."

With all due respect to Google, Eric Schmidt, and Sergay Brin, that statement almost sounds evil in and of itself.

I don't mean to flame folks here, but what if another company decided that their entire philosophy revolved around what their own CEO thought about things? What if this CEO was raised poorly, and without religion, and generally was a mean, racially prejucided man?

Google is an industry leader in the realm of Web search. They are moving into new technology areas as well, such as image searches, Usenet searches, product searches, directory searches, Web page caching, etc. What happens if Brin one day is the victim of a hate crime by a white person? Will he start blocking Google from indexing predominantly white Web sites such as J. Crew, Kuro5hin, or the New York Islanders home page?

It sounds good at first, but upon deeper exploration you'll see that it's:
a.) Childish
b.) Poorly thought-out
c.) Discriminatory
d.) A disservice to Internet users

I, for one, will no longer visit Google because I simply can't trust them anymore. I urge others to as well. AllTheWeb is nearly catching up to Google anyway, and is at least as good if not better.

Google is so much better, so why should Altavista survive in the long run?

While I agree with you on Google giving a much better service, I do believe there's a space for Altavista, Alltheweb and (hopefully) scores of other search engines as well. The reason is simple:- more than ads (television, pop-up or otherwise), it's search engines that uniquely determine how we browse the net. Sure, so far Google has *largely* been Good (tm), but that doesn't mean it will continue to be so. In particular, I'm concerned about the way results are arranged in Google (or any search engine); there's no accountability, nothing's open, there's only a vague comment about how The Algo gives PageRanks to each individual page. As we saw earlier, Google has taken results *without* publicly announcing that it's doing so.

Indeed, Alltheweb, in particular, sounds promising. It has more indexed documents with a faster "refresh cycle" than Google, a video, mp3, and a ftp search, and also says it can search through Flash movies. Of course, no way it can replace Google Groups, but all the same, it's definitely a viable alternative to Google. I believe we should welcome greater competition among search engines.

Free-market competition will help us avoid unduely relying on a single company. For Google's sake, I don't want it turn into a monopoly.

Google is so much better, so why should Altavista survive in the long run?

While I agree with you on Google giving a much better service, I do believe there's a space for Altavista, Alltheweb and (hopefully) scores of other search engines as well. The reason is simple:- more than ads (television, pop-up or otherwise), it's search engines that uniquely determine how we browse the net. Sure, so far Google has *largely* been Good (tm), but that doesn't mean it will continue to be so. In particular, I'm concerned about the way results are arranged in Google (or any search engine); there's no accountability, nothing's open, there's only a vague comment about how The Algo gives PageRanks to each individual page. As we saw earlier, Google has taken results *without* publicly announcing that it's doing so.

Indeed, Alltheweb, in particular, sounds promising. It has more indexed documents with a faster "refresh cycle" than Google, a video, mp3, and a ftp search, and also says it can search through Flash movies. Of course, no way it can replace Google Groups, but all the same, it's definitely a viable alternative to Google. I believe we should welcome greater competition among search engines.

Free-market competition will help us avoid unduely relying on a single company. For Google's sake, I don't want it turn into a monopoly.

Google is so much better, so why should Altavista survive in the long run?

While I agree with you on Google giving a much better service, I do believe there's a space for Altavista, Alltheweb and (hopefully) scores of other search engines as well. The reason is simple:- more than ads (television, pop-up or otherwise), it's search engines that uniquely determine how we browse the net. Sure, so far Google has *largely* been Good (tm), but that doesn't mean it will continue to be so. In particular, I'm concerned about the way results are arranged in Google (or any search engine); there's no accountability, nothing's open, there's only a vague comment about how The Algo gives PageRanks to each individual page. As we saw earlier, Google has taken results *without* publicly announcing that it's doing so.

Indeed, Alltheweb, in particular, sounds promising. It has more indexed documents with a faster "refresh cycle" than Google, a video, mp3, and a ftp search, and also says it can search through Flash movies. Of course, no way it can replace Google Groups, but all the same, it's definitely a viable alternative to Google. I believe we should welcome greater competition among search engines.

Free-market competition will help us avoid unduely relying on a single company. For Google's sake, I don't want it turn into a monopoly.

[Does anyone ever...] bother writing compliant html?

Oh yes there is choice. Try www.alltheweb.com and you will see they are quite close to google in quality. Sometimes they are better even.

Background: Among other things, I am always trying to discover music from independent (especially blues) artists that post mp3's of their stuff on the web. I have been boycotting the major record labels 100% for about 15 years (hooray for independents!) for several reasons: 1: CD prices have always been a rip-off, 2: most major label artists suck, 3: I have worked in music business (artist/bands/production) and detest the industry for the way that it exploits artists, 4: I have always loved discovering talented "unknowns" and turning other people on to them. I went through a new music dry spell until the web started to become a vehicle for independent artists to promote themselves.

It's amazing what's out there now - I've found great artists from all over the Americas, Europe, Australia, Russia and even a few from Asia. I have found a lot of crap, too. The mp3 search engines are essentially useless for this purpose (I don't want major label music) and I have never used Napster or any of the off-spring. Links pages are more often out of date than not and webrings have similar problems. I have contrived several search techniques that try to exploit the strengths of search engines and the likely information on an artist's site. One very simple one is to look for "mp3 +(insert name of a well-known blues standard) -(a lot of keywords to exclude the many sites that put "mp3" on every page that simply lists a song title just to pull in traffic) -(specific sites that pollute the searches)", to find artists that cover the song and also have their own tunes.

I have been a proponent of Google for many years. It came along just as I really started to dislike Altavista and I was an almost instant convert. But I am always on the lookout for a backup or something better. I have tried Teoma several times in the last year (as recently as last night), but I'm not terribly impressed. I find its interface and the way it presents results simplistic and dumbed down and it appears to have indexed far less of the web than Google. I got turned off Lycos years ago, when it seemed to want to become another portal/Yahoo (as if we need another one).

The one search engine that I do use as a regular alternative to Google is Alltheweb. For one thing, IMO, its advanced search is currently better than Google's (I swear that I have brought Google to its knees by entering too many keywords - it stops responding and is inaccessible for several minutes thereafter - this has happened several times). When I've done back to back comparisons with Google, Alltheweb seems to fare pretty well and seems to find more international pages than Google. The difference in top rankings can also be useful. Google has some nice features that Alltheweb does not, such as the elimination of duplicate pages.

For one-stop searching, I find Google best for me, but Alltheweb is a good alternative.

I use it if google gives me less than a dozen hits for a query.

Come on, Microsoft and bill gates have had bad terms associated with them at google for ages. Assuming that google changed this just for M$'s sake is ludicris.

If that was all they were worried about, they could simply have manually changed those searches to exclude MS (as they have done for people as small as Bernie Shiftman), that guy who spammed his resume around everywhere. Searches on his name would turn up pages bitching about him.)

Btw, you definitely deserve a +5 for plagiarizing this post verbatim. Well, except for the paragraph breaks, I guess.

I've had occasion to play around with AllTheWeb, which also often gives different (but still relevant) results on lots of queries. Worth a look.

.. I wish there were some good sites to read about Robert Frost

or if you want to take the road less travelled

Now they report that Google is announcing a service that AllTheWeb has offered for almost a year; that is: fresh news.

I'm a big fan of Google but this is old news. Really old news. It has been done already. Why is this more important than the Next Thing in web search? Because it's Google? Oh COME ON! Does Google pay you guys?

Here's the story from CNET.

The news page is nice, but the big thing is that google now searches "4,000 publications around the world. Previously, the site had searched 150 publications every hour."

Maybe now, Google execs won't have to publicly admit that All the Web ha[d] a better news search. ;-)

Several sites are running 9/11 front pages, notable Yahoo and Amazon.com. Some sites aren't, such as Google, some are more subtle such as Alltheweb.

Personally, I like the /. black banner. I suggest you click on it.

Also check out the Todd M Beamer Foundation. Make sure you have your credit card handy.

I'm very much an advocate of Google and have been for at least 3-4 years - I've turned many people on to it. Once you've tried Google, you'll never go back to Altavista (my favorite 5+ years ago, but it quickly started to suck).

But I have a major bone to pick with Google. I don't know if it is still the case, since the company where we experienced the following is no longer in business, but:

My old company registered a domain name 6 years ago. We built a web-site with contact info and proceeded to get all kinds of e-mails for products that weren't ours. It turns out that someone else had previously held the domain name and had gone tits-up, leaving a lot of unhappy customers out there. We were both in electronics but in radically different areas.

We tried ourselves to locate the other company so that we could point people to them, but only came up with a never answered phone number in Illinois. So we developed a stock polite response to these people wishing them luck and put a disclaimer on the web-site.

For the (300+) sites erroneously linking to our site where there was contact info, we sent polite e-mails asking them to remove the link, and got one of three results:

1) OK, will do - about 20%

2) No response - 70%.

3) F- you. - about 10% You wouldn't believe the arrogance of some people who felt that they had no duty to update their bad links. The most arrogant of all was a "webbastard" for a UK university site. He first claimed to have no such link and told us not to bother him - he had more important things to do (yes, he really said this). We sent him the page. He then told us that we should know better than to use a previously registered domain name. He portrayed tremendous ignorance of the living state of the web. We tried to explain to him (politely, in spite of his arrogance) that a domain is like a house - people move in, people move out. He finally changed the link but told us to f-off in the process (to which we sent him a polite thank you note).

The worst traffic sender was Yahoo directories (or whatever it's called) and it was impossible to get any Yahoo to do anything about it. There was no place to report this and we couldn't cram an explanation into a comment form that old took something stupid like 128 characters. We spent a lot of time trying to get Yahoo's attention, but they were an impenetrable monolith. We gave up on that.

Web half-life being what it is, things quieted down over time. But then a curious thing started happening. The e-mails from people looking for the old company started climbing again. We would ask these people how the got to our site and most were coming from Google. I was already aware of Google and using it by that time and pondered why, since Google seemed pretty "on-the-ball". Then I read about how their page-ranking works and it hit me. Google does not (or did not then) verify the links that they use for weighting. They simply weighted any link that they could find against a site. So the old links to the previous company's site, of which all pages but index.html were named differently than our pages (so, lots of 404's), were now being used by Google to point erroneously to our site. We sent mails to Google, but got nothing more than an acknowledgement. This seems like a major flaw in Google's logic. I don't know whether it was ever fixed or not.

As I say, I like Google and use it almost exclusively. I do use Alltheweb as a backup (their advanced search is better than Google's and allows more terms). I glad to see that Google got rid of the "XYZ" is a common word crap as well - when I do a literal search, words like "a, the, by" matter to me.

Google is quite good but Google is most definitely not perfect.

I just checked All the Web and Yahoo and they have sponsored links. So maybe it is more than just Google doing this.

Well, the story link is taking forever to load, so while I wait to read it I tell you people bout the new search engine I found:

http://www.alltheweb.com

Not as fast as google here, but returned me some good links. It makes paid links go on a "sponsored links" session just like google.

Heh, this is silly.... I can already think of possible abuse (don't click here) [alltheweb.com]

You'd think people would learn how to use a search engine!

"Google Sucks" on AlltheWeb returned 278 mathces - plus you can click on a category 'The Google Sucks Saga Continues'

"Teoma Sucks" on AlltheWeb returned 10 resutls, not too popular - but can click on the category 'Google Kicks Ass, Teoma sucks'

Guess size DOES matter occasionally. Plus, AlltheWeb is based on Free BSD- doesn't sell out like Teoma.

And does Google have MP3's, software downloads, and catagorization? Didn't think so. But pity the person who actually suggests going to someone other than the Google goliath.

"Google Sucks" on AlltheWeb returned 278 mathces - plus you can click on a category 'The Google Sucks Saga Continues'

"Teoma Sucks" on AlltheWeb returned 10 resutls, not too popular - but can click on the category 'Google Kicks Ass, Teoma sucks'

Guess size DOES matter occasionally. Plus, AlltheWeb is based on Free BSD- doesn't sell out like Teoma.

And does Google have MP3's, software downloads, and catagorization? Didn't think so. But pity the person who actually suggests going to someone other than the Google goliath.

But I can find lots on Bush and Hitler!

FTP Search

> BTW - Google Rocks! I never use anything else anymore!

I used to say the same 'til very recently. Alltheweb.com has good results too -- I suggest you try them out. They claim their index size is bigger as well: a fact I can attest to: I have found pages on alltheweb I haven't found on google (try this on alltheweb and google to see what I mean -- though I guess this isn't a _very_ good example :-)

Google still rules though (size isn't everything!) because of its relevancy rankings etc.

But what if the search engine business really is unprofitable? Would you pay to keep something like Google around? If so, how much, and how would you pay?

I hope that this is not a redundate post - will mozilla ever get the famous google toolbar. (For the poor no galeon in reach win32 mozilla people...)

And what is he thinking about AllTheWeb?

hey dudes! I just found a link on how you can make money selling junk! just click here to find out all about it!

• AllTheWeb
• Google
• Teoma
• WiseNut
• AltaVista

Norwegian search engine AlltheWeb on Monday declared that it indexes more Internet information than longtime pacesetter Google.

Then how come the word with the most search results (FYI: the) on Google, returns less results on alltheweb?

Google Pr0n Search finds 46,200 results.

Searching for pr0n via alltheweb.com leads to 2318 more potential pieces of pr0n to be seen.

Danny.

They have another advantage. Their news headline search is top notch.

Alltheweb's claims are not unfounded, and I find it always worth checking when google fails.

Here is one of several real life cases where it found software for me that google didn't.

(It still does, and google still doesn't.)

It seems that all the web includes multiple listings of the same directory on my site. I run a php enabled webserver so it tries a bunch of directories with ?M=D or ?A=S and then procedes to list them. Click here for an example of this. It also lists every single sub directory in all 5 or 6 subdomains that I host. Google is much better at not going into sub directories and just giving the main site. Google is by far a better choice when looking for main pages and not having to filter through all the /whatever/ directories.

I'll just say this:

Google manage to get a graph of the slashdot effect among the first 20 hits, while AllTheWeb just manage to get Cliff showing a Think Unix book (in weirdo hawaiian clothes). :-)

I don't know about you, but Google give me more relevant matches as usual. ;-)

Anyone remember archie ?

Yay my name is top on both AlltheWeb, and Google - does that mean I'm a slut? ;)

• Steve Kemp on AllTheWeb
• Steve Kemp on Google

Fun stuff.. bored now

I just tried to pull up one of my own pages with this engine. Got:

"Redirection limit for this URL exceeded. Unable to load the requested page."

Which, as near as I can tell, is their way of throttling commercial hits. Wonderful. Moving the mouse over the link doesn't reveal the address in the bottom bar, either, so the only way I can think of to obtain the address of the item it matches is by right-clicking and selecting 'copy link address', opening a new window and pasting it it (and having a browser that is capable of doing this), then editing the URL so only the target link text remains.

You can't even right-lick and open in a new window to do this. If you try, you get "about:blank" which, afaik, means they're using javascript.

These people sure go through a lot pains to render a result and then not let you anywhere near it. Saying they're bigger than Google is a bit like someone bragging about how their PDP-11 is bigger than my Athlon. Cripes.

I like the Image Search better than google's.

Jackson

You can actually download the songs, really nice.
I wonder if they will get in trouble from the MPAA?

AllTheWeb saves the day. Semicid is a vaginal contraceptive that's chalk full of Nonoxynol-9 to kill all those little swimmers dead. It's been around since 1973, so it shoulda been around when you went to college.

whois macthis.org

Domain Name: MACTHIS.ORG
Registrar: DOTSTER, INC.
Whois Server: whois.dotster.com
Referral URL: http://www.dotster.com/help/whois
Name Server: NS1.LUGGAGEPLANET.COM
Name Server: NS1.KNITE.NET
Updated Date: 05-nov-2001

Yes, the do exist. My guess is they are sooo tiny that even their name servers got /.ed

Also, Google is not the end to all queries. Don't forget alltheweb.

I couldn't agree more. Moderators: Why haven't this post been modded up?
Others: Give AllTheWeb a try!

"because the domains are different, the many thousands of links these sites have to one another all count toward the automated calculation of their popularity and quality at Google"

Wrong. PageRank counts links, whether they're on the same domain or not.

"giving them all a boost in the rankings and hence bringing Webseed more traffic and hence more revenue"

Wrong. Webseed tripped Google's spam penalty and their hub has had a PageRank of exactly zero since about January 25.

"AltaVista appears to be making a comeback"

Wrong. AltaVista is on its last legs. Fast Search is a quick, comprehensive search engine with advanced features. Inktomi is a good referrer for site owners because it powers big sites like MSN Search. Google is a big, quick search engine that's iextremely popular, very easy to use and it still has a habbit of putting the better sites near the top (even though people are trying very hard to spam it).

Calum
--
Calum I Mac Leod, Scottish Borders

I'm astonished. I use google for generic searches, but any time I need a specific answer, google is the one I definitely would not use, as it never returns the link I want in the first 3 pages.

So I have a list of twenty-something search engines I use for specific purposes as they all have their sweet spot.

Here are my top 7:

ask.com
altavista.com
findlaw.com
lycos.com
metacrawler.com
alexa.com
alltheweb.com
etc etc

Actually Google isn't the only search-engine using these techniques (that is: rank sites after how many pages that link to them). I have been on some lectures with FAST which Lycos and others are running their searches on. Perhaps they were first though, I don't know.

The main benefit Google has these days, is that they have ~8000 PCs clustered which they run the searches on, while FAST (as an example) has only 600. Google can therefore take the freedom to do searches that cost more processingpower, while others have to think of smart techniques to maintain good results without using the same power.

One example is that of searching for patterns, ie. several words in given order ("to be or not to be"). While Google uses their searchpower to find all those words, FAST saves all three words following each other ("to be or", "be or not", ...). This means three times the diskspace, but disk is cheap. This way, they have fast lookups, and save plenty of time.

Just don't visit sites that do Paid Inclusion, or realize that some results may be "tainted". Personally, I still find that All The Web still gives decent results for most things. This is because, it seems like All The Web & Google have this thing about who can get the most pages indexed.

For a good list of what search engines show what ads in what ways, check out this page at Search Engine Watch.