Slashdot Mirror

Search Engine Watch says that Google now performs an estimated 80% of the searches (200 million) on the Internet every day.

Something hasn't added up here yet. If Google gets 13% of so of search visits and we know who powers whom, how do we get to 80%?

Search Engine Watch says that Google now performs an estimated 80% of the searches (200 million) on the Internet every day.

Something hasn't added up here yet. If Google gets 13% of so of search visits and we know who powers whom, how do we get to 80%?

We use it to keep track of certain search terms and it works just great. It has advanced search options, as well as RSS Feeds and Trackback. Google Alert seems to have received a lot of recent press coverage, including this recent article in SearchEngineWatch.

• Surely not the ads.

Sure, anyone can type google.com into their browser, but for the 90% of the population who don't understand how the web works, pressing the Search button on their browser is the only option. The fact that Microsoft's search is getting better doesn't change anything though, as search.msn.com is already the IE default, and those people will be using that.

I wasn't aware that you needed to download special software to run this Google search application.

I think the application comes into picture via the Google Toolbar and also the need to somehow organize all the Google Services & Tools. & Google has also gotten into one-click Blogging via Blogger.

In addition there are tools that visually organize the Google Search results, SearchDay - Visualizing the Web with Google - 8 January 2003

When you start having a book called Google Hacks , you know that there are a lot of HPI's (like API's but for H-Hacking), you know that there is a better way to offer access to these hacks via well organized tools. That is the form and function of the application.

Of course there are other applications like Copernic ( a longer listing here Search Tools), but I think the current applications have miniscule following. What will come from Microsoft or Google will flood the market.

Google does have flaws too! Take a look at the googlewashing of miserable failure it brings up the official biography of George Bush as the first link. More coverage of this issue can be found on Searchenginewatch.com.

Two major search engines can't share space for very long, no matter how amiable it might seem. I was actually tired of both, myself. I have been frequenting Search Engine Watch for something more up-to-date (and less ad-sponsored). But then again, I guess I'm just doing that whole "underground rules" thing.

...they're going to have to really pick up their index size if they hope to compete with Google and ATW on finding anything that isn't on a site's home page.

And ATW's bots belong to Overture.

PS: I just can't resist making a little political observation: Yahoo! knows what they're doing. They're run by people who understand that a businessman's fundamental resonsibility is to ensure that gross income exceeds gross expenditures.

Google, on the other hand, is run by one Eric Schmidt, who believes that a businessman's fundamental responsiblity is to give blow jobs to Elton John, who in turn gives blow jobs to The Inventor of the Internet [TIOTI] himself.

Hint to investors the world over: If you see a once proud corporation with Eric Schmidt at the helm [Novell, anyone?], then short it.

And don't think for a second that the timing of this Yahoo!/Google announcement was a coincidence. There's no way in hell the boyz at Yahoo! were gonna let Schmidt get his grubby socialist paws on $4 Billion in IPO capital if they could do anything about it.

PPS: Overture had purchased both ATW and Altavista, so now Yahoo! owns Inktomi, ATW, and Altavista. Together the three of them might not equal Google's market share, but that's a lot of history, a lot of technology, and a lot of brainpower that Yahoo! has very quietly amassed.

PPPS: And it's all gonna run on FreeBSD! THE GPL SUX DONKEY DIX! LONG LIVE THE ANTI-LICENSE LICENSE!!

I've posted below, but in case it's lost in the noise, here's another one - open source search, open source biz plan:

Mobilemaps.com

It's a location search rather than a traditional search, and the demo is of California. Some commentary from Search Engine Watch is here.

The biz plan is geotargeted advertising - articles on it published here , and here.

Here's one - open source search, open source biz plan:

Mobilemaps.com

It's a location search rather than a traditional search, and the demo is of California. Some commentary from Search Engine Watch is here.

The biz plan is geotargeted advertising - articles on it published here , and here.

I know p2p search is hopeless, but here's some ideas on how to do it anyways. I'll phrase it like an inductive proof: first make a node, then add a neighbor.

NODE - I'd use Lucene. Lucene is a traditional keyword search engine that is fast, lean, free and open. It's carried under the Apache Jakarta project, so it's not going anywhere. And, it's easy to develop with. Alternatively, any good search will do... you could probably bang something together with GNU shell utils.

NEIGHBOR - Turn search into a common TCP/IP protocol, a la SMTP, FTP, etc.. Telnet to port 534268 (the digits that most look like "SEARCH"), and have something like this:

client: QRY p2p search efforts
server: HITS 1023
client: RETR 0
server: HIT http://searchenginewatch.com/sereport/article.php/ 2163581
...

If there are no results at that node, the server forwards you on:

client: QRY p2p search efforts
server: FWD 255.168.1.303

So, you'd start by querying your own host's search-engine. Perhaps it would spider N-deep from what you browse, so it would perhaps have ready responses for many of your queries.

But your own node may not have the answer for you, so you forward on to the next. How does the forwarding table get setup? One way to do it would be by hand, but also, I imagine posting "known expert" lists to gnutella could help automate the process. A list would be a map of keywords to IPs. These lists wouldn't need to be too robust, as they'd serve to occasionally seed the network, not constantly sustain it.

Once you had a good forwarding table on your node, you'd have access to quite a large search DB. With 100 nodes in the search network, each using 1GB for its index, and 3:10 index to indexed ratio, that's 100*1GB*3.3=330GB of indexed text. Let's say the average webpage is 100KB (?), that's a total search DB size of 3.4M pages. Increase the number of nodes to 10,000 and increase each node's index size to 10GB, and you have 3,460,300,800 pages, which is just about equal to Google, which is currently at 3,307,998,701. 10k nodes happens to be about what distributed.net is running right now, and 10GB is getting cheaper by the minute. ;)

As with Inktomi, Teoma has no free Add URL page -- but also as with Inktomi, Teoma crawls the web, so if you have links pointing at your web site, you may get included naturally.

Check it out here.

While everyone should have already heard of google, it's kinda dumb to use one search engine when you can use a meta-engine like Turbo10 that uses all of the main search engines and some lesser known...

Of course, AllTheWeb is giving Google a run for its money...in the race to make it to 4 billion pages indexed, so Google may fall back down for a while...

However, I don't think many ppl will switch because of a few thousand pages...

Actually, according to SearchEngineWatch, it was nearer 76% in May 2003, but point taken.

People have been defecting from using Yahoo as their primary search engine for years, and they're not about to come back unless Yahoo can offer search results that are comparable to Google.

Yahoo's search results are (currently) provided by Google, and have been since 2000.

They've been outsourcing and not using their own technology since at least 1996.

More info can be found here

The question is, 18 months from now, will I be able to find the Glade2 tutorial ? Only if you bookmark it, and much more easily if you publish your bookmarks.

I forgot to mention a couple of things.

First: Free-for-all link farms and crap have brought Google and the other engines to a point where they have to consider them spam and ban sites listed in them. They have had problems with these link farms polluting the results. One of the ways they identify a link farm is by seeing a lot of links on a page that are unrelated. So your Bookmarks page (and your Guestbook) could actually damage the sites listed instead of helping them. For more information, you'll have to go to Search Engine Watch and dig. But it's there.

The second thing is, you can use the "site:" command on Google (and some other engines) to limit your search to one specific site. So if you can just remember where the glade2 tutorial was, you should be able to find it after google adds it to its database.

nothing about grub here, but personally i really like this web site that have a few search engines on it: http://freddo.netfirms.com/. It also refers to Fravia's new website and his invaluable forum.

A good reference about search engines is also Search Engine Watch

have fun...

I don't understand what the big deal is. If the Chinese want to use Google, they will. If they want to use the new Government organized search engine, they will. The one that acquires most users wins. Open competition at its best. You should cheer competition in the marketplace.

Unless, of course, the Chinese governments artificially changes the odds by blocking Google or Google's Chinese partner. That may or may not happen. To my knowledge they are not blocking Google at the moment.

As much as I like Google, it has a monopoly on non-suckiness of search engines. If China's search can compete, unfairly or fairly, it won't be a mere arms race - only good can come of this.

Please don't abuse the word monopoly, it has a strong meaning. It doesn't just mean the lack of competition, it implies the exploitation of its position to prevent and nulltify competition.

See this for a definition.

And comptetions it has many. Inktomi, FAST, Teoma, Direct Hit, to name a few general crawler-based search engines.

See this for all the search engines there are out there.

Google has no monopoly whatsoever, it's just plain popular. it's popular because it's good at what it does, and the competitions just simply aren't as good at it.

Ironically, monopoly is what the Chinese government trying to achieve. The whole move is obviously politically motivated, the Chinese Search Alliance is government-backed, and undoubtfully the Chinese gorvernment will exploit its power to ensure the Chinese Search Alliance popularity and undermind Google's.

Why are all these people here dilluting the move into some technological arguement? Did the Chinese Search Alliance present any brillant search algorithm? No. So how do they intend to compete with Google? Market manipulation via their Government backing, duh!

who actually mixes them them the actual search results? NO ONE. no search engine of any size does this period. just because you say "they" do, and don't say who "they" are, doesnt make it true.

In the latest chapter of Google protecting their trademark, they even asked the dictionary folks at Wordspy to change their definition of the word "google" to prevent it from becoming a generic word. All this has caused mixed reactions and lots of news coverage by microdocs (formerly Google Village), Search Engine Watch, and Internet.com. Their latest target seems to be the Google Web APIs-based automated search service Googlert, who changed their name to "Google Alert" and explain that they were asked "politely" and have been "sympathetic" to Google's concerns. All this recent activity might be in the spirit of shoring up the Google brand and business image before an IPO...

In the latest chapter of Google protecting their trademark, they even asked the dictionary folks at Wordspy to change their definition of the word "google" to prevent it from becoming a generic word. All this has caused mixed reactions and lots of news coverage by microdocs (formerly Google Village), Search Engine Watch, and Internet.com. Their latest target seems to be the Google Web APIs-based automated search service Googlert, who changed their name to "Google Alert" and explain that they were asked "politely" and have been "sympathetic" to Google's concerns. It's nice to see that they let them keep the word 'Google' in the name - I guess Google is trying to keep web developers on its side.

Well, this may be the situation right now, but if it continues for long, someone is bound to start indexing faster and will therefore gain market share from Google/Fast...

I currently prefer Google, but both are fairly good...

Google will likely stay up there for a long time, but sooner or later it will be #2...

The barriers to entry for the search engine market are still not as high as in the auto industry (for instance)...

Maybe the next one will be Teoma (Owned by Ask Jeeves, Inc.)...

But until then, I'll still keep an eye on Search Engine Watch...

More information here

<A HREF>

Larry and others at google has said this in the past. Although I can't find proof on Google's web site (darn lousy search engine they use ;-), I did find this in an article on SearchEngineWorld:

Google examines link structures all over the web. By doing so, it can give every page a popularity rating known as "PageRank" (named after Google cofounder Larry Page). When you do a search, URLs with high PageRanks are more likely to be listed first. However, this will only happen if the pages also match other criteria, such as containing your search terms or being identified as being relevant to your search terms by analyzing the context of links.

According to this article, it was originally called "BackRub":

Google began as a search engine called BackRub. It was so named for what was its, (at the time), unique ability to analyze the "back links" pointing to and from a given website as part of its algorithms to search results. This approach to link analysis gained BackRub a growing reputation among those who had seen the technology. Today this technology is know as Google's patented "PageRanks" technology.

Another reference: http://www.eyrie.org/~zednenem/2002/08/30/

This page at Search Engine Watch has a half dozen or so real-time search query viewers. It also has some quasi-realtime "search popularity" stuff too.

Yes but was it against SearchKing specifically or any site that did what SearchKing did?

Well, pointed out in a previous article, Google has kept the exact method of its page rank for this as well as all other cases on the lowdown. However, if link farms were getting downgraded, I would wager that such things as Scientology would start to drop as well. Then again, I just did do a search for Scientology and noticed quite a few highly ranked sites critical of Scientology. You'll note that scientology.org ranks very highly, but I believe that a domain that nearly exactly matches a search ranks very high for non-common terms. For instance, try searching for "searchking recipes". They're #1. Now, try searching for "recipes". They're still there, but they're in the 80s.

Though, my ultimately answer is: I don't know. But judging by the catastrophic drop of their pages, I would wager they just manually set the PageRank for any SearchKing site to a low value, or capped it similarly. Further, depending on how you read the wording in their reply, you could assume they are implying that:

And Search King admitted that Google had the right to take action in response, including changing Google's opinion of the importance of the Search King site by changing the PageRank assigned to that site.

Granted, I'm sure any strict logicican (and certainly Google's lawyers), would suggest that this simply means Google has a right to lower anyone's Page Rank either directly or indirectly through modification of their ranking algorithms, and it doesn't even state that Google changed its algorithms at all, let alone specifically to affect SearchKing.

Inktomi's current customers

Yahoo would be well-served building a cross-reference ranking from Google + Inktomi's results. Most of my searches are quite pointed anyway though, so I'm not sure how this could be improved.

Go try the Hotbot or MSN searches yerself. This may well be the future rankings on Yahoo results.

As a trial, I searched for "Oklahoma Dry Spell" and although there was one coinciding match in the top 2, the rest were completely different. It seems Inktomi is a bit more relaxed for inclusions. (14,888 vs Yahoo's 12,800).

For one of the myriad of search engine reviews comparing (roughly) Inktomi and Yahoo/Google, see this page

mug

When HotBot debuted in May 1996, it gained a strong following among serious searchers for the quality and comprehensiveness of its crawler-based results, which were provided by Inktomi, at the time. It also caught the attention of experienced web users and techies, especially for the unusual colors and interface it continues to sport today.

HotBot gained some notoriety when it switched over to using Direct Hit's "clickthrough" results for its main listings in 1999 (see the Using Direct Hit Popularity Results page for more about this). Direct Hit was then one of the "hot" search engines that had recently appeared. Unfortunately, the quality of Direct Hit's results couldn't match those of another "hot" player that had debuted at the same time, Google. HotBot's popularity began to drop.

Even worse, HotBot also suffered by being owned by Lycos (now Terra Lycos). Lycos had acquired HotBot when it purchased Wired Digital in October 1998. Lycos failed to make search a priority on its flagship Lycos site as well as HotBot through much of 1999 and 2000, as it focused instead on adding "portal" features. The company refocused on search in late 2001, making significant improvements to the Lycos site (described above). HotBot's chance at redemption is supposed to come in late 2002. Watch this space!

When HotBot debuted in May 1996, it gained a strong following among serious searchers for the quality and comprehensiveness of its crawler-based results, which were provided by Inktomi, at the time. It also caught the attention of experienced web users and techies, especially for the unusual colors and interface it continues to sport today.

HotBot gained some notoriety when it switched over to using Direct Hit's "clickthrough" results for its main listings in 1999 (see the Using Direct Hit Popularity Results page for more about this). Direct Hit was then one of the "hot" search engines that had recently appeared. Unfortunately, the quality of Direct Hit's results couldn't match those of another "hot" player that had debuted at the same time, Google. HotBot's popularity began to drop.

Even worse, HotBot also suffered by being owned by Lycos (now Terra Lycos). Lycos had acquired HotBot when it purchased Wired Digital in October 1998. Lycos failed to make search a priority on its flagship Lycos site as well as HotBot through much of 1999 and 2000, as it focused instead on adding "portal" features. The company refocused on search in late 2001, making significant improvements to the Lycos site (described above). HotBot's chance at redemption is supposed to come in late 2002. Watch this space!

One of the other search engines (Altavista?) used to let you see 20 random queries

It was Excite, but they stopped in 2001.

Google is so much better, so why should Altavista survive in the long run?

While I agree with you on Google giving a much better service, I do believe there's a space for Altavista, Alltheweb and (hopefully) scores of other search engines as well. The reason is simple:- more than ads (television, pop-up or otherwise), it's search engines that uniquely determine how we browse the net. Sure, so far Google has *largely* been Good (tm), but that doesn't mean it will continue to be so. In particular, I'm concerned about the way results are arranged in Google (or any search engine); there's no accountability, nothing's open, there's only a vague comment about how The Algo gives PageRanks to each individual page. As we saw earlier, Google has taken results *without* publicly announcing that it's doing so.

Indeed, Alltheweb, in particular, sounds promising. It has more indexed documents with a faster "refresh cycle" than Google, a video, mp3, and a ftp search, and also says it can search through Flash movies. Of course, no way it can replace Google Groups, but all the same, it's definitely a viable alternative to Google. I believe we should welcome greater competition among search engines.

Free-market competition will help us avoid unduely relying on a single company. For Google's sake, I don't want it turn into a monopoly.

<meta name="keywords" content="metatags are evil, metatags must die, death to the meta tags">

<meta name="description" content="If you can read this meta description tag, then the author's wish for the end of metatags has not yet come true. Someday, it will.">

And, less funny, the second story:

<meta name="GENERATOR" content="Microsoft FrontPage 5.0">
<meta name="ProgId" content="FrontPage.Editor.Document">
<meta name="author" content="Danny Sullivan">
<meta name="channel" content="E-Commerce/Marketing Channel">
<meta name="description" content="Now supported by only one major crawler-based search engine -- Inktomi -- the value of adding meta keywords tags to pages seems little worth the time. ">

It's always been a confusing issue for site owners. Should I use commas between words in the tag or not? How many times can I repeat a word on the page without getting banned? If I don't list a term in the tag, does that mean my page won't show up?

"The meta keywords value is just one of many factors in our ranking equation, and we've never given too much weight to it. That said, we will continue to use it as long as our relevance modeling shows that it adds value," said Ken Norton, director of product marketing for Inktomi's web search division.

These types of links are called deep links .

There has already been quite a lot of controversy regarding deep links, dating all the way back to 1999.

In fact, one major free website hosting company, whose name escapes me at the moment, does not allow you to deep link to their members' pages. Instead, you are forced to go to that member's home page first (I imagine that they are checking for referers or some such thing).

Clearly, deep linking is beneficial, but some companies just don't get it.

And I conjecture that, Google being low on the advertising and high on the usefulness, it is more popular among Linux users (being *slightly* more knowledgeable than the average user). Thus the proportion of Linux visitors to Google is way above the true proportion of Linux users on the Internet.

And I conjecture that, Google being low on the advertising and high on the usefulness, it is more popular among Linux users (being *slightly* more knowledgeable than the average user). Thus the proportion of Linux visitors to Google is way above the true proportion of Linux users on the Internet.

If, indeed, Google is ranking links by IP address then it's a brutally flawed concept to begin with

I didn't say Google is "ranking links by IP", I said it's confusing 2 sites that have occupied the same IP address. In fact, Google is doing all it's normal ranking procedures (text analysis and link analysis), but screwing up the very last step by associating the rank with the wrong URL. Yes, this is a big error, but it's easy to spot: If you click on a Google link, and the site you find is about what Google says it's about, this problem hasn't affected your search.

I actually doubt the problem is entirely Google's fault. Very few people have reported this problem, and whenever I've tried to help them, they turned out to be "strictly end-user" types who couldn't tell me anything useful about their server configuration. Therefore, I haven't been able to exclude the possibility that this Google "error" is prompted by misconfigured web servers.

You would be shocked as some of the silly misconfigurations enacted by commercial web hosting companies. For example:

Apparently a bunch of hosting companies have decided "404 Not Found" errors are obsolete, and started returning "403 Forbidden" responses when browsers/robots requested non-existant files from the web host customers. Unfortunately (for their customers) Googlebot interpreted those respones differently when it comes to robots.txt. "404" meant "no restrictions, come on in", while "403" meant "stay the hell out". So a bunch of customers who didn't know anything about robots.txt (and shouldn't need to) suddenly got their sites kicked from Google, because their hosting company confused Googlebot.

(Google has, in fact, recently changed their policy on 403 errors because of mistakes like this.)

There has also been a roundup story about Scientology v. Google at Search Engine Watch. Possibly before Google started directing censored results to Chilling Effects. This is of interest to SEOs and, more generally, to those who are curious about search engines.

The story today is in the New York Times. Essentially the same story, but the audience is different. PHBs might read the NYT. Academics read the NYT. It is a mainstream publication with a reputation for quality.

Yet another attempt to whitewash online reality by eliminating not only websites but also their very mention (especialloy in search engines). Somewhat reminiscent of Scientology's heavy-handed attempts at search engine censorship employing the DMCA?

From the Infoworld article: "Even if the pages no longer exist on XS4ALL sites, we want the search engines to remove the link because it still advertises a handbook for destruction. People will start looking for it elsewhere and we don't want that."

If you do a search for Deutsche Bahn on Google, the first two links are now to this news story. Precisely the opposite effect?

SearchDay - Curling Up with a Good Book Search Engine.

Did ask.com buy out teoma?

It isn't too hard to follow the link labeled Press Information at the Teoma site to find another link to the Search Engine Watch report entitled Ask Jeeves Acquires Teoma from Ovtober, 2001.

The good folks at Teoma were even nice enough to excerpt the following:

"Ask Jeeves has purchased the Teoma search engine, which has attracted interest over recent months as a potential relevancy challenger to Google."

You may even notice that Ask Jeeves is plastered all over the contact page. I don't think they're hiding the connection between the two brands from anyone.

Has the use of search engines impaired our ability to follow links from one document to the next?

Heck, a Google search of your exact question led to the NewsTrove tracking of the assimilation. Then again, the other results were a little iffy. ;)

Did ask.com buy out teoma?

It isn't too hard to follow the link labeled Press Information at the Teoma site to find another link to the Search Engine Watch report entitled Ask Jeeves Acquires Teoma from Ovtober, 2001.

The good folks at Teoma were even nice enough to excerpt the following:

"Ask Jeeves has purchased the Teoma search engine, which has attracted interest over recent months as a potential relevancy challenger to Google."

You may even notice that Ask Jeeves is plastered all over the contact page. I don't think they're hiding the connection between the two brands from anyone.

Has the use of search engines impaired our ability to follow links from one document to the next?

Heck, a Google search of your exact question led to the NewsTrove tracking of the assimilation. Then again, the other results were a little iffy. ;)

Considering that most major search engines now place links according to payment, it's a short step to turning the browser, or the whole OS into Bonzi Buddy.

Not that I would mind if the OS did some contextual search for me to bring up results while I'm working, but I've seen enough ads for the X10 wireless camera, thank you.

Considering that most major search engines now place links according to payment, it's a short step to turning the browser, or the whole OS into Bonzi Buddy.

Not that I would mind if the OS did some contextual search for me to bring up results while I'm working, but I've seen enough ads for the X10 wireless camera, thank you.

Half of Google's revenue comes from selling text-based ads
According to this Sept 2001 article, 2/3 of Google's revenue is from advertisements.

Google can't survive without ads, but it's ironic considering the founders Brin and Page once said "...we expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers...[A]dvertising causes enough mixed incentives that it is crucial to have a competitive search engine that is...in the academic realm."

Just don't visit sites that do Paid Inclusion, or realize that some results may be "tainted". Personally, I still find that All The Web still gives decent results for most things. This is because, it seems like All The Web & Google have this thing about who can get the most pages indexed.

For a good list of what search engines show what ads in what ways, check out this page at Search Engine Watch.