Google Blocks 'Optimized' Pages

Rhett Creighton writes "For the past few years, webmasters have found tricks that bring their page higher for a given keyphrase search. Google recently implemented a filter to block sites that appeared to be tricking it into gaining a higher ranking. This NYTimes article reports of angry retailers who are losing their businesses, while this article gives more technical conspiracy theories of what google is actually doing."

Comments

Seth F's theories

[turg]

Seth Finkelstein has been posting a few theories lately on what Google is up to. (Also contains links to other articles.) He suspects they are using some sort of Bayesian filtering around the rule "If a simple search has spam-related keywords, penalize high-spam-scoring results" (spam being search-keyword spam on web pages -- not e-mail spam)

Re:Seth F's theories

[scrytch]

Seth Finkelstein has been posting a few theories lately on what Google is up to. (Also contains links to other articles.) He suspects they are using some sort of Bayesian filtering around the rule "If a simple search has spam-related keywords, penalize high-spam-scoring results" (spam being search-keyword spam on web pages -- not e-mail spam)

Easy to defeat a bayesian filter: use a sentence generator. Feed a few hundred mission statements and "about us" pages into a markov model and let it churn out babble. You're not really concerned with being 100% coherent, since none of your generated spam is actually on the site having its ranking pumped up. You just want uniqueness, the bane of any bayesian filter.

Needs works

[grub]

Google needs to fine tune their code. Enter "goatse.cx" and clicking "I'm Feeling Lucky" brings you to their uptime page at Netcraft, not the horrible image we all know and cherish.

ps: f1st pr0st.

Re:Needs works

[ComaVN]

Reading "goatse.cx" and "I'm Feeling Lucky" in one sentence sent shivers down my spine.

Re:Needs works

[Channard]

Enter "goatse.cx" and clicking "I'm Feeling Lucky" brings you to their uptime page at Netcraft, not the horrible image we all know and cherish.

The key words are 'I'm Feeling Lucky' - not 'I'm in the mood to be mentally scarred for fricking life.' Plus, it'd stop you getting a gruesome thumbnail if you typed in 'goats' in images.google.com.

Re:Needs works

[Anonymous Coward]

The answer can be found for your problem by
visiting http://goatse.cx/robots.txt

If you don't care for visiting the site ( I wonder why)
you will see they don't want any bots in the site.

Re:Needs works

[rasjani]

But if you enter just plain "goatse" and hit enter, you'll get a google directory for Scientology =)) See here

The sky is NOT falling.

[mmoncur]

Every month or so, Google updates its database again, and every time, webmasters all over the world whose pages happened to go lower in the rankings complain that Google is broken and the sky is falling. This time is no different, except that mainstream news has picked up the story. Here are a few facts to keep in mind:

1. You can't say with authority that "Google has implemented a filter." Google isn't talking about how their rankings work. The webmasters and SEO types are like astronomers trying to figure out how Google works by observing samples of results. Take everything they say as a theory and nothing more.

2. There's a fine line between making responsibly search-optimized pages and spamming Google, and many of the people who complain are on the spamming side of that line. If you look in the forums where SEO types (and spammers) hang out, 90% of the messages are complaining that their site has disappeared and Google is wrong. If you look in web development forums, 90% of the messages are from people excited to see their pages' position increase.

3. For every webmaster that complains about their site's Google position going down, there are one or more sites whose positions have gone up. Often they're equally deserving of the traffic.

4. There are strong rumors (and some statements from a Google representative) that suggest that this is the last major update to Google's database, and that incremental "freshbot" updates will continue from now on. If this is the case, it may only be a day or a week before your site changes position again, so why complain?

5. Most importantly, notice that it's always webmasters complaining. Never end-users. Guess which group Google considers its customers?

Re:The sky is NOT falling.

[Peyna]

The only people google really needs to keep happy to stay in business are the people using the search engine to find things, and the people who pay to have a text ad on the side of the page.

Since these people are a small subset of actual users, and probably are not paying for an ad; I doubt there is any concern at all about how they feel.

Re:The sky is NOT falling.

[Stephen]

Every month or so, Google updates its database again, and every time, webmasters all over the world whose pages happened to go lower in the rankings complain that Google is broken and the sky is falling. This time is no different.

I agree it's no different from usual, in some ways. But on the other hand, there have been an increasing number of such stories recently. Of course, the loudest complaints come from those whose sites have sunk. But many ordinary people feel that Google search results have worsened gradually over the last few months, and the press is right to pick up on this.

My personal theory, albeit based on no hard evidence, is that Google has started to rely less on PageRank. Newspaper articles usually blame people subverting PageRank in some way, but I think they're wrong. Those papers don't realise that Google uses a combination of many methods to rank pages, not just PageRank.

What I think happened is this:

1. Google invented PageRank. It's not perfect, but it worked much better than anything else, and they took over the world. It's also really hard to thwart: just linking to your own sites doesn't improve your PageRank; you need high-quality links from third parties.
2. Unfortunately, PageRank didn't work so well when blogs came along, because of their high amount of interlinking. So Google was forced to reduce the weight of PageRank in the algorithm.
3. The problem is that this meant they were now depending more on traditional ("SEO") metrics, which are more easily manipulated.
4. So now they're chasing round trying to catch the cheats, while allowing legitimate sites through. But as the history of search engines before Google shows, this is a losing battle, and there will be complaints from site owners who have been wrongly demoted, as well as from users who are seeing more crud.

As I say, all this is speculation, but it makes sense to me.

Re:The sky is NOT falling.

[Webmoth]

"Guess which group Google considers its customers?"

It's not webmasters. It's not end-users. It's advertisers.

Advertisers are the only ones that Google has to answer to. If they do something that makes their advertisers go away, you can bet that they will quickly reverse that decision. And, the only thing that will make advertisers go away is whatever makes end-users go away.

You see, for Google (of for any other media outlet, for that matter) the advertiser is the customer, and the end-user or reader is the product. The content is the means of delivering the product to the customer.

What defines "customer?" Someone who gives you money in exchange for goods and services.

Google does have customers other than advertisers: select webmasters who purchase Google's services for their own Intra-/Internet presences. Even here, the customer is not the end-user, but is the webmaster himself. In this case, Google's best interest is to return searches the webmaster considers favorable (which, ultimately, are those pages the webmaster thinks the end-user should see).

So, you see, Google's interests are where the money is. And the money is in advertising and select webmasters. Perhaps in Google's Internet search, they favor companies who have purchased their services. Perhaps they demote companies who have refused to, but that's only speculation on my part.

Re:The sky is NOT falling.

[czei]

While most months you would be right, in Nov. 2003 you are not. The update to Google this particular month featured a radical change in the algorithms used to rank sites. In previous months for the past several years the rankings would change little every month-- if you look at the top 10 sites for a particular search term you'd see some sites move up a few levels, and other sites move down a couple of rankings.

This time, however, from what I can estimate most of the topped ranked pages have changed for a particular search term, which indicates a major change in Google's algorithm, which is particularly newsworthy. If that isn't news, then what is?

The reason businesses are complaining is that there are tens of thousands of small businesses that make their living from customers who find them via Google, which is also newsworthy. And most aren't SEO professionals or scammers who've illictly tried to artificially boost their rankings, thank you very much.

Whether the change will also elicit complaints from searchers has yet to be seen-- some search terms seem to return relevant results and some do not.

Re:The sky is NOT falling.

[sphealey]

Unfortunately, PageRank didn't work so well when blogs came along, because of their high amount of interlinking. So Google was forced to reduce the weight of PageRank in the algorithm

I think Pagerank and similar algorithms worked just fine when blogs came along - they correctly signaled a potential trend away from historical media control patterns to a new way of disseminating information - particularly political information. But the entities which have historically made a lot of money by controlling the flow of information don't like that, so they have been pressuring Google and other search engines very hard to "eliminate" blogs from search results. Thus returning to the status quo ante.

sPh

Re:The sky is NOT falling.

[karlk79]

What came first the chicken or the egg. I agree thats where they get the money but you have to have an audience to get the ad people to spend. I believe it begins with the user. You have to keep them.

Re:The sky is NOT falling.

[LostCluster]

It's not a pressure from the media elite that says blogs don't belong high ranked in Google, it's users. Blogs are great at telling Google what articles in other publications are most authoritative on a topic, but a "blog" is by definition not one. (Of course, blog software can be used to run an authoritiative site... but that's a different category all together.)

Blogs got highly rated because groups of friends linked to each other's blogs. However, those sites shouldn't be linked that high for that reason alone. So, if the only external links on a site lead in circles, then the site really isn't that good, and it gets bumped down.

Basicially, the idea of "I'll link to you if you link to me, and we'll both move up in Google!" now does more harm than good.

Re:The sky is NOT falling.

[karlandtanya]

Advertisers are the only ones that Google has to answer to.

I disagree.

There are 2 parts to this business transaction you describe--that is, Google selling ads to folks for money.

The ads, per se, have no value. What the people who pay Google the money really want is for people to see those ads. The "product" if you will allow an overused term--is eyeballs.

Google needs to maintain its position as the place to go to find things on the web. That means making sure that the vast majority of surfers say "Google is your friend.", not "Google links to spam."

Junk Sites

[Oen_Seneg]

If this stops the hundred odd spam sites coming up every time in now what seems to be every time I search fo a query that involves a dictionary word, then as a veteran web surfer, I'm all for it. But there'll always be a way around the filter, take it a day, a week, a month, a year. Things really can only get worse.

2nd Article Text (In case of /.'ing)

[Trolling4Columbine]

With the huge number of postings on all the various forums, concerning this update, most people don't know where to start looking for information about the recent Google update. The following is an attempt to put down rationally (I hope) most of the information that is known and the (unproven) theories behind the update algo.

Introduction.

Starting on the 16th of November, a major shift in results was seen on Google. Veterans recognised that Google appeared to be doing a major update, not seen for many months, as reported first on WebMasterWorld who named it Florida, continuing the tradition of naming updates rather like hurricanes. In this case it was a hurricane! As was usual with many updates, there were moans and groans as people complained about their sites falling. Many people were unaffected (including us) but the symptoms of the sites being dropped were not usual. No penalties, such as PR0, seem to have been applied against pages that had fallen - yet none of the pages targeted at specific key phrases, particularly index/home pages, appeared in the top results for these search terms. Indeed some had dropped hundreds of places and, in some cases reported, off the scale. Yet these pages did appear for obscure phrases and were obviously still in the index.

It appeared to us and to several other respected names (though hotly disputed by others) that some sort of over-SEOd filter had been applied to check if overt SEO had been done for that particular phrase. It was as if Google were checking to see if external links to the site included the phrase, on-page optimisation was being done for the phrase and even if the domain included the phrase. If the density of the optimisation, both on and off the page, appeared too artificial, then a filter was tripped and down went the page - solely for that phrase.

Google had never looked favourably on abuse of their systems and many established SEOs looked upon this algo tweak as a way of Google getting rid of the abuses of links and stopping the scrambling for getting (and sometimes buying) links including your required anchor text from other high PR, but probably irrelevant to your subject, sites. It seemed to make sense.

On Friday, 21st November, Google decided to tighten the filter. All hell broke loose as tens of thousands of sites disappeared from positions they had held (in some cases) for years. We noticed some of our client sites plummeting for their major key phrase from being #1 to total invisibility. Yet this was only in highly competitive areas, not for their secondary phrases. These sites were, in most cases, not highly optimised, had not sought reciprocal links but had achieved their rankings through being on the web for 4 or 5 years. The bad news was that their company name and domain included the key phrase, sites (including directories) linking to those sites included the key phrase in their links and Google interpreted this as over-optimisation and down they plunged. In many areas all the top 20 ranking sites disappeared, including industry leaders, to be replaced by educational sites, news review sites, government sites, major shopping portals or directories. Something major had happened - but what?

The Facts!

Thousands of web pages have been suddenly demoted in the Google search results, primarily on the main commercial search terms for which they targeted their pages to be replaced by other sites who, in the main, referred to the search term obliquely. Several were the main shopping portals or business directories which gave listings for companies who may provide the services requested, many were not.

Very high-ranking authority sites seemed to be unfiltered.

The changes were starkly obvious on regional English language Googles where a regional filter was employed and there were less commercial sites with authority.

An example for Google UK is the search for the word shelving. On the

Good for Google

[bahamat]

I for one really don't give a rip if retailers throw a hissy over this. When I search the web it's because I want information, not because I want to buy something.

If I want to buy something I use Froogle. That's what it's there for.

The Real Moral: Google is not your ad agency

[Thag]

The real moral here is that if you're depending on your placement in a search engine for free advertising, you'd better have a backup plan.

Jon Acheson

Re:The Real Moral: Google is not your ad agency

[Slider451]

Exactly right.

Analogy: If somebody you barely know started giving you a dollar a day for no reason whatsoever, then a couple years later stopped giving you the dollar, would you be pissed at the giver or be thankful for the generosity you did receive?

These companies act like they're owed something more based solely on the fact that they were getting it before. Merit-less entitlement. I bet the company owners aren't welfare fans, yet that seems to be what they're arguing for here.

Re:The Real Moral: Google is not your ad agency

[Kurt Gray]

Absolutely agree. I do a little web site consulting on the side and I usually tell my clients three bits of advice for better exposure:

1. Think about putting some kind of unique, useful, and/or entertaining content on your web site that people will want to visit, link to from their own web site, and even email to their friends. Good content builds traffic.

2. Take basic steps to make your pages search engine friendly. Descriptive titles, simple honest meta-tags, useful text in every page, descriptive links to other pages, etc.

3. Don't be obsessed with your Google ranking. Don't give money to anyone who claims they can boost your Google ranking. If you want to spend money for traffic than buy Google Ad Words or sponsor links, at least that way you pay per actual click-through rather than paying into a bidding war for an uncertain better ranking.

Re:What am I missing?

[Triumph The Insult C]

did you read the article?

this is an instance where one company has fouled up the search results. google's policies state to not do that, and if you do, you may be removed from search results.

Oh, no!!!

[armando_wall]

Google was my #1 tool to find my penis enlargement products.
Now I can't even get a home loan!!! And I can't consolidate my debts!!!

What am I gonna do???

Re:Oh, no!!!

[kurosawdust]

get an email account?

Re:Oh, no!!!

[arkanes]

Post your email address on usenet, of course.

I LIKE GOOGLE. :)

[wo1verin3]

I wrote a small software app for a the company I work for, and someone linked to it from a discussion forum.

That gave that name of that program xxxxx.exe 1 hit on google. About 3 days later a search for the xxxxx.exe provided 3 hits, two of them were porn sites that somehow harvested the name of our .exe file.

While it's not a huge deal, I e-mailed Google and heard nothing for 4 days. I didn't expect a response and told them that in my e-mail. However then I recieved a personalized (not a form) e-mail regarding my comments and that they'd take the issue seriously.

24 hours later they were able to filter out these porn sites that were harvesting new terms that appeared in Google.

I gotta say props out to the boys there, it's one classy establishment.

Re:I LIKE GOOGLE. :)

[jos3000]

That's ironic because xxxxx.exe is what I've called my porn website generating program. :-)

Re:What am I missing?

[turg]

Google tries to find a formula that gets you the best result for what you're searching for. Some web site owners try to figure out this formula in order to make their page show up in the search resuls when it shouldn't (e.g. by having words in the URL or within certain tags on the page -- rather than by having content relevant to that topic). This makes Google less useful (including for the purpose you describe) and so Google is "demoting" pages which show signs of using these tricks. This tug-of-war has been going on as long as there have been internet search engines. The difference now is that Google accounts for so much of the searches on the net that getting a lower rank in Google can have a huge effect on a site's traffic and so people freak out about it.

Example of what google is trying to prevent

[Anonymous Coward]

Olagam

Re:Example of what google is trying to prevent

[bigman2003]

As well they should.

I HATE when I am trying to search for something, and just keep coming up with crap sites.

I do web design, and my customers keep asking me how to get further up in Google rankings. I always tell them the same thing- have good content, and get other legitimate sites to link to you.

Some of them have been using these services that set up the link farms, and I will be very happy when it goes away.

I would much rather have the REAL website be the basis for the ranking, not a bunch of crap.

Fine By Me!

[dukeluke]

Google - thanks abound from me. I personally find it distasteful when I'm searching for research on a particular topic. More times than not - most of the top listings are by an amazon or other shopping portal that has NOTHING to do with my search.

Yes, many businesses are being hurt by GOOGLE's policy - however, it is GOOGLE's search engine! They have done nothing wrong but try and give authentic results to their Web Surfing friends.

Then use Froogle

[Chibi Merrow]

Google has a specialized tool to access their search engine specifically to do shopping/price comparison... So yes, you are missing something. :)

Besides, these sites were using hacks to artificially inflate their pagerank instead of providing a higher quality site to increase it.

Good.

[Space cowboy]

Given the underlying reason why google is a good search engine (leveraging the popularity of the site by others), I don't want "my" search engine to be fooled into giving me commercially-orientated results.

If Google has re-organised the page-ranking system to cut out the link-merchants, I give it an unreserved thumbs up :-)

Simon.

Reliance on Google...

[mopslik]

...is no substitute for a business plan.

So some people are trying to cheat the system, and Google is taking steps to prevent this. Good for them, I say. I'm tired of getting pages that appear to be legitimate, only to find that they're just redirect fillers.

As for Google's practices in general, retailers are free to moan and groan about their rankings, but there is no obligation for Google to specifically cater to their needs. If Google decided to change its algorithms, such that all links were turned alphabetically rather than by PageRank, they would be well within their rights to do so. Of course, I imagine that such a move would result in many people seeking other search engines soon enough.

Re:What am I missing?

[worm eater]

About 50% of the time when I'm searching, I AM looking for vendors of a product in order to do price comparisons

Google already has a search engine specifically designed for price comparisons... maybe that's what you're missing.

It's called Froogle.

I'm not sure why they haven't added a tab for this on their main page, as it would make a lot of sense to separate out commerce-related searches from information-related searches while making both easily accessible.

Re:What am I missing?

[FesterDaFelcher]

Google will not be hurting the "small, no-name individual", they will be hurting the companies that do nothing but set up spam-filled door pages for products and services that have nothing to do with what you are searching for.

Complaining webmasters?

[daBass]

Don't they realize that Google wasn't created for them? Rather, it is created for surfers. There is one surefire way to get noted on Google if your business depends on it: advertise. You get what you pay for.

Re:Complaining webmasters?

[microcars]

I'm always amazed at how people/companies will spend ungodly amounts of time/money on something just to avoid paying a "fee".

In this case, to avoid paying Google for an Ad Word fee, they spend piles of money and time setting up cross-linking websites hoping to "beat the system".

This is the same mentality that alot of people who tried to scam DirecTV have/had. I know people who spent MORE MONEY on Hacked DirecTV cards (that constantly needed to be updated...) than it would have cost to have purchased a full-blown monthly subscription!
All they got was Bragging Rights that they "beat the system".
"Look", one of my friends would say, "I'm getting EVERY CHANNEL FOR FREE!". uh huh. not anymore.

hmmm, I had some sort of a point with this post, but now I've lost it. sorry.

misleading writeup

[taybin]

The writeup made it sound as if the NY Times article was about upset retailers mad at google when really they're upset at the company that was tricking the search results.

Re:What am I missing?

[Quixote]

Maybe I'm missing something....

"Common sense" comes to mind.

If you read the NYT article, it clearly says that some unscrupulous vendors were clogging up the search results. So, on the first page of Google results, you'd get most of the sites from the same vendor (shell sites, put up specifically to increase the number of links between them, thereby increasing the PageRank).

Google is trying to level the playing field, so that no one site can dominate the results.

Looking at your complaint, I think it would make sense for Google to have a "vendors" checkbox, which would list sites selling stuff, as opposed to sites giving out information.

I did a google

[theMerovingian]

for "boost page ranking", and got this cheesy online marketing company:
http://www.page-rank.boost-web-site-traffic.com/

My first reaction was that these guys are scam artists. But, they did appear at the top of the page...

Obviously it is still possible to scam google. To web experts: how do they do it?

Waaaaaa

[scorp1us]

The free search engine that listed me for free is no longer paying off! Waaaaaa! Waaaaaa!

I for one welcome the change. Too many times have 20 of the top 30 links taken you two one site, but camafloged to google somehow as to look seperate. I experieced this painfully while looking for ringtones for my cellphone.

Google is first and foremost a search engine, not a marketing tool. Those who thought otherwise are finding out they are sorely mistaken.

Re:Waaaaaa

[jafac]

Google is first and foremost a search engine, not a marketing tool. Those who thought otherwise are finding out they are sorely mistaken.

That said:
having been a net surfer since 1993, and having NEVER EVER EVER paid money for an online service, other than my ISP -
I would pay Google a subscription fee monthly, maybe even something like $5, to ENSURE that it remained a tool to serve web users, instead of a tool to serve site operators.

Google is THAT important to me.

Without Google, I believe that the web, as a resource for information and communication, would become pointless within the space of 5 years.

I still don't think I'd pay money for any other site. (Including slashdot). Many many sites have started free, then initiated a "premium service" or subscription fee - and faded into obscurity. I think that only Google could survive such a move.

Re:What am I missing?

[arkanes]

I think that'd be cool, or at least have an option for it. Googling for information about things like video cards is almost impossible because all the links are to stores.

All your searches are belong to google

[Ba3r]

Despite my personal pride in being one who tries to grasp a concept/issue via multiple sources from different perspectives, i just realized that the vast majority of my information these days funnels through google. And i know i am not alone.

I would wager google's potential control of information distribution and content filtering rivals that of major centralized information outlets like CNN or the NY times. Kinda unnerving.

Expectations too high?

[mopslik]

From the second linked article:

An example for Google UK is the search for the word "shelving"... On the main Google search for the same phrase, the results return 1 site that sells shelving, 6 shopping portals, 2 Universities and 1 Amazon store. Yet previously these results showed 9 shelving suppliers.

What does this guy expect? He searches on a single word and expects that every result be a retailer? Why not add some extra terms, like "buy" or "seller" or "retail" after that, buddy?

Seriously, should I start crying foul when I search Google for "dog" and it returns information on breeds rather than specific pet-stores?

Re:What am I missing?

[stevesliva]

So, if Google turns their search engine into a search engine that ignores those types of search results then they've just moved out of the No.1 position in my favorite search engine list. Maybe I'm missing something....

I think you're missing the fact that these are the sites that are basically screaming, "I am RELEVANT. I am so so so relevant! relevantrelevantrelevant! I am so relevant that I have to do several sneaky things to show my relevance! Look at me! memememe!" They're the hyperactive seven year-olds of online retail, and they're all Amazon (or whatever) affiliates that are selling you the same things for the same price. Why you'd demand to see them, I don't know.

What Google needs

[fleener]

Google needs to separate commercial pages from purely informational pages. Anyone searching for information (not sales products) gets inundated with e-commerce sites. It's a waste of time, building complex queries that weed out dominant company names. affiliate sites, and words like "cart."

Google needs to expand its advanced search options to include toggles for different ranking criteria. Anyone who has searched in vain knows this. I have several dead-end searches every week.

Google needs to change it's outdated automatic e-mail reply blurb. Staff may read every e-mail received but saying "[we] try to send personal responses to each message" is just baloney. That was true in the early years.

Google needs to get off its laurels and start listening to its customers again.

Re:Source Claims SCO Will Sue Google

[paiute]

And then:

Microsoft releases MSN Music Club
SCO sues Apple

Microsoft releases Sparkle
SCO sues Macromedia

Microsoft ships new Xbox model
SCO sues Sony

Gates farts
SCO sues the dog

Barry Llyod - Agent to the Stars

[teamhasnoi]

Google's results seem to have degraded over the last couple of years. Full of linkfarm websites and searches that seem to lead to information, but only return another (sponsored) 'search'. It has made it a pain to find common information as the results are spammed by Amazon affiliates, and hard to find obscure information as the results are often buried under a pile of non-appropriate links.

I run a teeny Miva Merchant site which used to be a 'regular' (html, cgi) web store. It has been up since 1997 with little or no changes. No meta tags or keywords on the site.

A week after I messed with that stuff, Google put my site at the top of the list when you do a search for some uncommon keywords related to it. It was nice to see, but so far of limited usefulness.

Now with the Miva site (which are notorious for not being indexed) I will have to come up with a revised strategy.

I would tell anyone - pay attention to your tags, and the immediate content of your site. Everyone is fighting for placement using similar keywords, so checkout the top results and see what they are using.

OT - Doesn't Barry look like he's making some deal on the phone? "Yeah, I can get you Ted, but he's gonna cost ya. He's huge at the Laugh 'n' Snort. I think he'll go for that, I'll call ya back."

Google no longer works

[fermion]

It seems that google's biggest problem is still the family of link farms that combine incestuous connections and pages full of keywords to get hits of their ads. The problem has gotten so bad lately that I tend to ignore the first few hits. I use the URL and displayed text to try and guess which hit might contain useful information, and not just indexes of links to other pages of indexes to other pages of indexes...

I suppose it is just another symptom of monoculture. It would be real nice to have two or three search engines that were reliable and shared the market space. OTOH, it is a 'free' service, so I am not complaining, and am happy to have such a service.

Google just doesn't care.

[actionfront]

As a long time user of Google and a reasonably large advertiser - our company is now questoning whether Google will survive the next couple of years. Through contacts, we have pointed out to Google (and submitted spam reports and submitted poor results reports) that one of our competitors has 2,700 duplicate doorway entry pages to their site. Several hundred of those are illegally indexed using "our" trademarked name. We also advised them of another competitor with 159,000 doorway pages - all indexed and showing up in results. Google's response . . . (silence)

Re:What am I missing?

[Triumph The Insult C]

i don't think you did. if you did, you would have known that they gave a prime example of a larger company fouling up the results and really hurting a small company that had been doing fine before the larger company started spamming google.

case in point, last night, i decided i was going to buy the "i'm a bomb technician. if you see me running, try to catch up" shirt. when i searched for it, the first 7 results were more or less the same page (different bgcolors, different rotating ads, different popups), which all pointed to the same url for checkout. i didn't find the vendor i wanted until the 2nd page ... how many people do you expect to actually go to the 2nd page?

i guess ihbt.

"I'm Feeling Lucky"

[siskbc]

Reading "goatse.cx" and "I'm Feeling Lucky" in one sentence sent shivers down my spine.

Not really. "I'm feeling lucky" that I, unlike the gentleman in the picture, do not have a 6" diameter asshole.

Re:"I'm Feeling Lucky"

[Tackhead]

> > Reading "goatse.cx" and "I'm Feeling Lucky" in one sentence sent shivers down my spine.
>
> Not really. "I'm feeling lucky" that I, unlike the gentleman in the picture, do not have a 6" diameter asshole.

At least! I know what to give thanks for tomorrow at the family dinner! Dude, Thanks!

This is great!

[gunnk]

A few days ago I tried a seach for "cellular customer satisfaction". The first several pages were bogus resellers (many of them the same page under different URL's). None of them contained the kind of information I needed about how customers rate the various cellular service providers. This morning the same search is yielding lots of useful data instead of the fake spam-like pages I had been getting.

KUDOS to Google for fixing this! Whatever changes they've made to their pagerank algorithm, Google is suddenly working again like I expect.

PageRank was never that simple

[LostCluster]

Google's "PageRank" formula is their top-secret way of determining in which order to display websites for any given keyword. Everyone knows that refering links is the main component of PageRank, but Google has always been hush-hush as to what else is included in the formula.

It's also known that PageRank isn't a static formula. Google reserves the right to change it at any time, in what is known by Google-watchers as a "Google Dance".

The only legit way to be highly ranked by Google is to be the most authoritative source for information about whatever you discuss, and naturally links will form from other quality websites on your topic and up the PageRank scale you go. www.microsoft.com being a 10/10 ranking doesn't indicate that Google likes Microsoft, it just simply indicates that site is the most authoritative site about a topic a lot of people talk about, Microsoft's products.

Any other way to cheat the system will result in penalties applied to your score. It's not so much a filter as it is negative factors in the formula. Google steadfastly claims that it doesn't maintain a blacklist of "bad" sites, but it is clear that sites designed to cheat Google's PageRank formula always fail once Google tweaks the formula. They don't need a blacklist, they simply identify the characteristics that define a "link farm" and then apply a penality. If a given site has a lot of links to external domains, very little non-link content, and absoulutely every linked to site returns a link back to the orignal site, it sure smells like a link farm and that's what the system penalizes.

To put it bluntly, anybody who's business depends on being displayed on the first page of Google results should be buying AdWords placements. If you're working hard to stay #1 in the editorial results, you're never gonna win. And no, just because your business depends on it doesn't mean you get to sue when Google pulls an ill-gotten #1 ranking out from under you.

This freakin' attitude of entitlement

[karlandtanya]

Really becomes annoying.

Google is a service that is, for the most part, free to those who benefit from it.

Somebody discovers that they can manipulate this service to increase their benefit.

The people who provide this (free) service chose to ignore those manipulations. Maybe they deliberately lower the ranking of some pages, to hear the whiny TFH crowd speak.

Then those same whiners--who contributed NOTHING to the process from which they benefit--scream for damages.

If someone invented a pill to make people immortal and one of these jerks didn't get his pill, these same folks would want the inventor jailed for murder.

Until you form a union and negotiate a contract with google--that includes a "past practices" clause, just STFU.

Re:What am I missing?

[YouHaveSnail]

Right, but when you search through Google (or any search engine) you expect to find rankings that accurately reflect the relevance of the page. If you search on "testosterone" you probably do not want the first ten pages of links to be to "Joe's Patented Penis Enlargement System" just because hardworking Joe set up dozens of shell web sites solely to increase his site's Google rank.

If Joe (or any other web site owner) really wants to use Google as an advertising medium, he ought to pay for a sponsored link and be done with it. Joe has no right to manipulate the ranking system, and if he's going to do that he ought to be prepared to suffer the consequences.

prime example: ringingphone.com

[EvilStein]

I was looking for ringtones for my Ericsson phone a few weeks ago, and (literally) the first 3 pages were all sham sites that were filled with ads and lead back to - you guessed it - www.ringingphone.com!

Those sham sites are no longer showing up in the Google results, and I can actually find what I was looking for instead of having to wade through the cesspool shopping mall that the internet is turning into.

Re:What am I missing?

[uchian]

Read the article. The company set up dozens of websites in order to get themselves in the top ranks of google - as in, all of the top 10 that you would see on google would in actual fact *be the same company*.

That acn hardly be considered a situation in which you can price-compare. Google is simply fixing the problem.

Coincidence? Or IPO Strategy . . .

[mjt AG]

Isn't it a wierd coincidence that they change their ranking algorithm before the end of the quarter, when people are supposed to spend the most money during the holiday shopping?

Another wierd coincidence - by the end of the next quarter, they're supposed to go public (Google IPO - oogles of $$$).

Another wierd coincidence - With Yahoo's recent acquirement of Overture, and the ownership of multiple search technologies, it is rumored that they may end their contract with Google soon, and power their own search.

Another wierd coincidence - MS "trying" to develop a better search engine, already trying to take on Google, even before Longhorn.

Now isn't that a great way to drastically increase their revenue prior to becoming public by making all the top searched commercial sites pay for ads on Google, especially when a bulk of those sites' revenue come from the upcoming holiday season?

Wow - Google seems to also know how to play chess!

Bayes Wars

[fm6]

Easy to defeat a bayesian filter: use a sentence generator.

Easy for you. I doubt if Ms. Gifts Baskets knows what a Bayesian filter is. This is an arms race between the index spammers and Google computer scientists -- and Google has a lot of really good CS people. Perhaps somebody smart enough could start a business defeating Googles filters, and distribute the costs among spammer clients. But it'd be cheaper just to buy a Google sponsored link.

That's why I have to laugh whenever I read stories speculating that Microsoft might do to Google what they did to Netscape. It's one thing to steal a big consulting/integration contract by throwing lots of marketing and engineering resources at the customer. But to dominate the search engine world, you have to earn and maintain the trust of millions of users who pound on your engine every single minute. I used to think that Infoseek, Altavista, and the others died solely from corporate neglect. That's partially true, but they were doomed anyway, as soon as Google appeared. Because none of them ever understood what Brin and company seem to understand instinctively -- a public search engine requires hard work on a huge scale, and it never stops.

This is good news

[LoRider]

It's great that Google continues to tweak there search engine to produce the most accurate results. The people complaining are the commercial businesses that are relying on Google search results for free advertising. First off, these "businesses" should not be relying solely on Google search result hits for traffic to their sites, they have to advertising somewhere. Secondly, Google has every right, and duty, to continue battling against businesses from gaming the search results.

My sites have probably increased in position on Google because of these changes, but I don't plan on reducing my advertising budget.

One Search Does Not Fit All...

[Anm]

It seems to me that the real source of the problem is in an assumption by Google, and most other search engines, that they can provide a purely string based search without any semantic context.

For the issue at hand, the new filtered results implicitly assume no site can legitimately grab too many links above some threshold for non-trademarked words. But in the case of a shelving provider being referenced as "shelving" (as exampled in the second article), that is not the case. The result is commercial entities with a high PageRank are filtered out.

This is fine for users looking up info about shelving, but not for user looking to buy shelving. Hence my comment on semantic context. In this case, a simple drop down of search prefixes ("I want info about...", "I want to buy...", ...) that select the a particular customized search algorithm in the Google engine. This proviudes enough info to direct the user to a customized algorithm tailored for the users expected type of results. In some respects, this is what Froogle and the other google specialty searches are about, except through the front page interface. This also provides a legitimate hook into bringing blogs back into the fold without interfering other users by looking for reviews and/or user comments.

And in traditional Google fashion, Google could provide links to possible alternative searches in other semantic contexts, just like they already do with spelling and the like.

The prefix approach is only one possibility. Maybe a sentence parser would be better (if you can convince current users to convert from keyword searches).