Google Blocks 'Optimized' Pages

Rhett Creighton writes "For the past few years, webmasters have found tricks that bring their page higher for a given keyphrase search. Google recently implemented a filter to block sites that appeared to be tricking it into gaining a higher ranking. This NYTimes article reports of angry retailers who are losing their businesses, while this article gives more technical conspiracy theories of what google is actually doing."

Comments

Seth F's theories

[turg]

Seth Finkelstein has been posting a few theories lately on what Google is up to. (Also contains links to other articles.) He suspects they are using some sort of Bayesian filtering around the rule "If a simple search has spam-related keywords, penalize high-spam-scoring results" (spam being search-keyword spam on web pages -- not e-mail spam)

Junk Sites

[Oen_Seneg]

If this stops the hundred odd spam sites coming up every time in now what seems to be every time I search fo a query that involves a dictionary word, then as a veteran web surfer, I'm all for it. But there'll always be a way around the filter, take it a day, a week, a month, a year. Things really can only get worse.

I LIKE GOOGLE. :)

[wo1verin3]

I wrote a small software app for a the company I work for, and someone linked to it from a discussion forum.

That gave that name of that program xxxxx.exe 1 hit on google. About 3 days later a search for the xxxxx.exe provided 3 hits, two of them were porn sites that somehow harvested the name of our .exe file.

While it's not a huge deal, I e-mailed Google and heard nothing for 4 days. I didn't expect a response and told them that in my e-mail. However then I recieved a personalized (not a form) e-mail regarding my comments and that they'd take the issue seriously.

24 hours later they were able to filter out these porn sites that were harvesting new terms that appeared in Google.

I gotta say props out to the boys there, it's one classy establishment.

Example of what google is trying to prevent

[Anonymous Coward]

Olagam

Re:What am I missing?

[arkanes]

Tell them to stop fucking with the search algorithms, then. I get plenty of hits for no-name stores when I'm looking for stuff.

Re:Needs works

[Anonymous Coward]

And by the looks of it, netcraft needs to fine-tune their code as well...
Unless goatse is *really* running MS/IIS on linux (?)

Waaaaaa

[scorp1us]

The free search engine that listed me for free is no longer paying off! Waaaaaa! Waaaaaa!

I for one welcome the change. Too many times have 20 of the top 30 links taken you two one site, but camafloged to google somehow as to look seperate. I experieced this painfully while looking for ringtones for my cellphone.

Google is first and foremost a search engine, not a marketing tool. Those who thought otherwise are finding out they are sorely mistaken.

Re:Waaaaaa

[jafac]

Google is first and foremost a search engine, not a marketing tool. Those who thought otherwise are finding out they are sorely mistaken.

That said:
having been a net surfer since 1993, and having NEVER EVER EVER paid money for an online service, other than my ISP -
I would pay Google a subscription fee monthly, maybe even something like $5, to ENSURE that it remained a tool to serve web users, instead of a tool to serve site operators.

Google is THAT important to me.

Without Google, I believe that the web, as a resource for information and communication, would become pointless within the space of 5 years.

I still don't think I'd pay money for any other site. (Including slashdot). Many many sites have started free, then initiated a "premium service" or subscription fee - and faded into obscurity. I think that only Google could survive such a move.

Re:Needs works

[evilpaul13]

The Goatse.cx page is proof to me that there is something to the claim about media desensitizing people. ;)

on the flip side

[SuperBanana]

On the flip side, I help run a set of mailing lists for car enthusiasts. We've been around for 12-13 years, and we have archives that cover almost all of that. We were using htdig, and it sucked/broke a lot, so we tried google's search, and it sorta worked ok(mailing list archives are horrible for google because of the crosslinking etc)

Until recently- the last year or so is when we started noticing problems. The last 6 months, complaints about holes and odd behavior have skyrocketed; for example, you can search for "master cylinder 2003" and get some posts, but search for "master cylinder" and 90% of the time, you won't find anything from 2003 in any of the results. The whole seems to be from about 2001-2003, and some messages simply can't be found.

I emailed google pointing out the problem, and after 2-3 weeks, got a long-walk-short-pier kind of email that basically said "we can't really control how much we index, sucks to be you". Thanks google.

Soon as we find a free, full-text search DB engine that doesn't suck, we're switching....we'll probably give htdig another shot, but it'd be nice to have something a little smarter.

What Google needs

[fleener]

Google needs to separate commercial pages from purely informational pages. Anyone searching for information (not sales products) gets inundated with e-commerce sites. It's a waste of time, building complex queries that weed out dominant company names. affiliate sites, and words like "cart."

Google needs to expand its advanced search options to include toggles for different ranking criteria. Anyone who has searched in vain knows this. I have several dead-end searches every week.

Google needs to change it's outdated automatic e-mail reply blurb. Staff may read every e-mail received but saying "[we] try to send personal responses to each message" is just baloney. That was true in the early years.

Google needs to get off its laurels and start listening to its customers again.

Google ain't that good

[Anonymous Coward]

I live in a small town in northern British Columbia called Dawson Creek.

Trying to search for a query like "Dawson Creek" information in your search query yields mostly hits for the television show under a similar name.

After politely emailing Google mentioning to them that although we're a small town of 10-20K people, we do like to have a web presence, nothing has changed.

Meanwhile, searching in Altavista for "Dawson Creek" information yields results only pertaining to the city. Hmm.

Could be google forcing people to pay for adwords.

[Phil John]

...think about it for a moment. Some searches return mainly academic institutions and not commercial entities. So google might have gone...

"Hmm...if peoples businesses are no longer on the first page what are they going to do? Bingo...they are going to pay for sponsored matches to stay in the game!".

And lo and behold...we have finally found out what should replace ???????? !

Step 1)Create worlds largest and most popular search engine
Step 2)Shaft lots of commercial sites that use the most searched for keywords, causing lots more people to purchase sponsored matches
Step 3)Profit!!!

As the article said, it seems to be only the most popular search terms. Which means probably those that require the highest price per click on their sponsored matches. Now, more people will try for sponsored matches on those keywords, pushing the price up (artificially) high.

We are always warned about the dangers of a monopoly/monoculture, and this is precisely why.

Good

[duffbeer703]

Every scumbag with a copy of Frontpage and $19.99 a month for web hosting is trying to become a millionaire with affilate commissions.

The more Google does to obscure the peddlers of obnoxious scams and "get rich quick" Amway types, the better.

Re:The sky is NOT falling.

[Stephen]

Every month or so, Google updates its database again, and every time, webmasters all over the world whose pages happened to go lower in the rankings complain that Google is broken and the sky is falling. This time is no different.

I agree it's no different from usual, in some ways. But on the other hand, there have been an increasing number of such stories recently. Of course, the loudest complaints come from those whose sites have sunk. But many ordinary people feel that Google search results have worsened gradually over the last few months, and the press is right to pick up on this.

My personal theory, albeit based on no hard evidence, is that Google has started to rely less on PageRank. Newspaper articles usually blame people subverting PageRank in some way, but I think they're wrong. Those papers don't realise that Google uses a combination of many methods to rank pages, not just PageRank.

What I think happened is this:

1. Google invented PageRank. It's not perfect, but it worked much better than anything else, and they took over the world. It's also really hard to thwart: just linking to your own sites doesn't improve your PageRank; you need high-quality links from third parties.
2. Unfortunately, PageRank didn't work so well when blogs came along, because of their high amount of interlinking. So Google was forced to reduce the weight of PageRank in the algorithm.
3. The problem is that this meant they were now depending more on traditional ("SEO") metrics, which are more easily manipulated.
4. So now they're chasing round trying to catch the cheats, while allowing legitimate sites through. But as the history of search engines before Google shows, this is a losing battle, and there will be complaints from site owners who have been wrongly demoted, as well as from users who are seeing more crud.

As I say, all this is speculation, but it makes sense to me.

Re:The sky is NOT falling.

[Webmoth]

"Guess which group Google considers its customers?"

It's not webmasters. It's not end-users. It's advertisers.

Advertisers are the only ones that Google has to answer to. If they do something that makes their advertisers go away, you can bet that they will quickly reverse that decision. And, the only thing that will make advertisers go away is whatever makes end-users go away.

You see, for Google (of for any other media outlet, for that matter) the advertiser is the customer, and the end-user or reader is the product. The content is the means of delivering the product to the customer.

What defines "customer?" Someone who gives you money in exchange for goods and services.

Google does have customers other than advertisers: select webmasters who purchase Google's services for their own Intra-/Internet presences. Even here, the customer is not the end-user, but is the webmaster himself. In this case, Google's best interest is to return searches the webmaster considers favorable (which, ultimately, are those pages the webmaster thinks the end-user should see).

So, you see, Google's interests are where the money is. And the money is in advertising and select webmasters. Perhaps in Google's Internet search, they favor companies who have purchased their services. Perhaps they demote companies who have refused to, but that's only speculation on my part.

Re:The sky is NOT falling.

[czei]

While most months you would be right, in Nov. 2003 you are not. The update to Google this particular month featured a radical change in the algorithms used to rank sites. In previous months for the past several years the rankings would change little every month-- if you look at the top 10 sites for a particular search term you'd see some sites move up a few levels, and other sites move down a couple of rankings.

This time, however, from what I can estimate most of the topped ranked pages have changed for a particular search term, which indicates a major change in Google's algorithm, which is particularly newsworthy. If that isn't news, then what is?

The reason businesses are complaining is that there are tens of thousands of small businesses that make their living from customers who find them via Google, which is also newsworthy. And most aren't SEO professionals or scammers who've illictly tried to artificially boost their rankings, thank you very much.

Whether the change will also elicit complaints from searchers has yet to be seen-- some search terms seem to return relevant results and some do not.

Barry Llyod - Agent to the Stars

[teamhasnoi]

Google's results seem to have degraded over the last couple of years. Full of linkfarm websites and searches that seem to lead to information, but only return another (sponsored) 'search'. It has made it a pain to find common information as the results are spammed by Amazon affiliates, and hard to find obscure information as the results are often buried under a pile of non-appropriate links.

I run a teeny Miva Merchant site which used to be a 'regular' (html, cgi) web store. It has been up since 1997 with little or no changes. No meta tags or keywords on the site.

A week after I messed with that stuff, Google put my site at the top of the list when you do a search for some uncommon keywords related to it. It was nice to see, but so far of limited usefulness.

Now with the Miva site (which are notorious for not being indexed) I will have to come up with a revised strategy.

I would tell anyone - pay attention to your tags, and the immediate content of your site. Everyone is fighting for placement using similar keywords, so checkout the top results and see what they are using.

OT - Doesn't Barry look like he's making some deal on the phone? "Yeah, I can get you Ted, but he's gonna cost ya. He's huge at the Laugh 'n' Snort. I think he'll go for that, I'll call ya back."

Google no longer works

[fermion]

It seems that google's biggest problem is still the family of link farms that combine incestuous connections and pages full of keywords to get hits of their ads. The problem has gotten so bad lately that I tend to ignore the first few hits. I use the URL and displayed text to try and guess which hit might contain useful information, and not just indexes of links to other pages of indexes to other pages of indexes...

I suppose it is just another symptom of monoculture. It would be real nice to have two or three search engines that were reliable and shared the market space. OTOH, it is a 'free' service, so I am not complaining, and am happy to have such a service.

Re:The sky is NOT falling.

[meta-monkey]

1) That much is true. Google doesn't talk much about how PageRank works, so there's nothing left for web designers to do but throw the chicken bones and glean the answers. That's what this guy's doing, and there's nothing wrong with that.

2) No, most of the people complaining are NOT on the spamming side. Read the article. It's primarily people on the COMMERCIAL side. Commercial sites have beend dropped, and educational (or entirely non-related sites) have taken their place.

3) Other sites have gone up, yes. Are they deserving of the traffic? No. When a search for "web design Calgary" returns the home page for a hockey team in the #1 spot, that's broken.

4) Those rumors have persisted for eight months, with no sign of being true or false.

5) Of course they're the ones complaining, because they're the only ones who would notice. End users aren't going to be able to tell the difference when something changes in the Google algorithm, because they're not searching for the same thing over and over again.

That all said, I'm not worried. My site didn't get hit this time, but it did once in the past, and it looked a lot like this for about two weeks. I (and others) complained LOUDLY on the forums, and with good reason. My site, and my competitors' sites, were dumped in favor of completely ridiculous and off-topic sites.

For instance, say you make widgets in Mytown. You would like it if, when people searched for "Mytown widget supplier" they got a link to your site somewhere. That's what my situation was. The front page was about 6 Mytown widget suppliers (with me at #2) and the other 4 or so were directories of widget suppliers. Then, the google update hits. Then, myself and all my competitors are gone, some of the directories are still there, and the rest of the results are nonsensical. Widget suppliers on the other side of the country. Some guy's blog about how his sisiter, who's visiting from Mytown, once bought a widget. Oh, and she's a big Air Supply fan. That's not an improvement. Now, as in your #5 point, are the end users likely to notice and complain? Probably not. They'll just figure there aren't any widget suppliers in Mytown. Are the webmasters going to complain? Hell yes, loudly, and rightly so.

Then, about two weeks later, things returned to normal, for the most part. The positions were all mixed up, and some new results appeared, and no, the sky did not fall. So, I would imagine that if the webmasters and SEO'ers would chill for a couple weeks and let the new algorithm propogate through Google, they'll find things the same as they every were--mostly.

Google just doesn't care.

[actionfront]

As a long time user of Google and a reasonably large advertiser - our company is now questoning whether Google will survive the next couple of years. Through contacts, we have pointed out to Google (and submitted spam reports and submitted poor results reports) that one of our competitors has 2,700 duplicate doorway entry pages to their site. Several hundred of those are illegally indexed using "our" trademarked name. We also advised them of another competitor with 159,000 doorway pages - all indexed and showing up in results. Google's response . . . (silence)

bagel calories search

[thbigr]

Do this search and you will NOT see a web site listed that actually tells you the calories in a Bagel. I think Google is doing the RIGHT thing,in fact they should do it even MORE:

Try apple calories, fruit calories, etc... same thing

Check this place out:

[Gannoc]

After suspecting I was being scammed by my mortgage broker, I types "Mortgage broker scams" into google. I'm in favor of google doing anything it needs to do to stop garbage like this.

This is the type of link I got back. The last paragraph is my favorite:

http://www.mortgage-broker-in-1.com/mortgage_bro ke r_scams.htm

The text of the page, that was followed by a LOT of links:

"Sourcing on the web for the best deals on mortgage broker scams? Well you've definitely arrived at the right place because that's what we're information experts in. Of course, being a new information web portal we don't yet have a monumental amount of information on the precise search term you were looking for - mortgage broker scams, but we're getting there.

Locating relevant and useful mortgage broker scams sites is often difficult. Which is why we created this website. Detailed research went into building this web site on mortgage broker scams to send you to the best web sites.

Coming across the best mortgage broker scams websites isn't as easy as it sounds. After a meeting of our team of planners and engineers we decided to build this site to assist you with your navigation. I'm thrilled to say that the countless hours of work we did studying info databases on mortgage broker scams for you to visit.

As the mushrooming of e-commerce continues mortgage broker scams businesses learn more in offering their products and services for sale The biggest benefit that the web mortgage broker scams businesses will maintain over store-front mortgage broker scams businesses is the significant savings they have an operating a successful business.

Re:Then use Froogle

[captainkibble]

Funny I did a search as you suggested for 'Beer' and another for 'Cadbury' and few of the links I followed had anything to do with a company in the United Kingdom. The few with any relation were for selling UK products to Americans. Many where infact for buying flags. My point was/is that Froogle is far less than a perfect (hell its not even passable) solution for people the world over so suggesting it as brilliant wonder solution for finding products and that people are stupid not to know of it is not helpful. And that is seemingly what people are doing here. Either that are Americans are obsessed with British Chocolate and Beer. ;) I have no doubt that once Google have finished their beta phase they will provide functionality to other countries. No doubt a tab will appear on the frontpage of google then too. In the meantime non-Americans might find it difficult to find products online. Yes I know there are other search engines but I have yet to find one quite as useful as google. Maybe altavista.co.uk might be a better choice from now on?

Re:What am I missing?

[YouHaveSnail]

Right, but when you search through Google (or any search engine) you expect to find rankings that accurately reflect the relevance of the page. If you search on "testosterone" you probably do not want the first ten pages of links to be to "Joe's Patented Penis Enlargement System" just because hardworking Joe set up dozens of shell web sites solely to increase his site's Google rank.

If Joe (or any other web site owner) really wants to use Google as an advertising medium, he ought to pay for a sponsored link and be done with it. Joe has no right to manipulate the ranking system, and if he's going to do that he ought to be prepared to suffer the consequences.

Re:The Real Moral: Google is not your ad agency

[CommieOverlord]

Most people would be pissed for not giving that extra dollar a day anymore. It's one of those things.

There was a study done (maybe a few) which rated customers' satisfaction with some service (a restaurant I believe). The customers were quite happy for time. Then one day they got an extra freebie, and their satisfaction went up. But the next day when they didn't get that freebie, the satisfaction plummeted below the original level despite receiving the same service. And the satisfaction level stayed low for a long time.

I imagaine the same is true for that extra a day. Once people have expectations they're disappointed, even if their expectation were unreasonable.

google responses!

[Ender Ryan]

Heh, I emailed google once regarding scientology-related censorship. It was one of those times when scientology was pressuring google to remove some pages from their index, and I just sent a polite email voicing my opinion about the situation.

I didn't expect a reply, but several days later I got a personal response from someone at google explaining what they were going to do about the problem and agreeing with what I said.

Indeed, a very classy establishment, especially when compared to other U.S.-based companies these days...

Coincidence? Or IPO Strategy . . .

[mjt AG]

Isn't it a wierd coincidence that they change their ranking algorithm before the end of the quarter, when people are supposed to spend the most money during the holiday shopping?

Another wierd coincidence - by the end of the next quarter, they're supposed to go public (Google IPO - oogles of $$$).

Another wierd coincidence - With Yahoo's recent acquirement of Overture, and the ownership of multiple search technologies, it is rumored that they may end their contract with Google soon, and power their own search.

Another wierd coincidence - MS "trying" to develop a better search engine, already trying to take on Google, even before Longhorn.

Now isn't that a great way to drastically increase their revenue prior to becoming public by making all the top searched commercial sites pay for ads on Google, especially when a bulk of those sites' revenue come from the upcoming holiday season?

Wow - Google seems to also know how to play chess!

Bayes Wars

[fm6]

Easy to defeat a bayesian filter: use a sentence generator.

Easy for you. I doubt if Ms. Gifts Baskets knows what a Bayesian filter is. This is an arms race between the index spammers and Google computer scientists -- and Google has a lot of really good CS people. Perhaps somebody smart enough could start a business defeating Googles filters, and distribute the costs among spammer clients. But it'd be cheaper just to buy a Google sponsored link.

That's why I have to laugh whenever I read stories speculating that Microsoft might do to Google what they did to Netscape. It's one thing to steal a big consulting/integration contract by throwing lots of marketing and engineering resources at the customer. But to dominate the search engine world, you have to earn and maintain the trust of millions of users who pound on your engine every single minute. I used to think that Infoseek, Altavista, and the others died solely from corporate neglect. That's partially true, but they were doomed anyway, as soon as Google appeared. Because none of them ever understood what Brin and company seem to understand instinctively -- a public search engine requires hard work on a huge scale, and it never stops.

Re:Bayes Wars

[Zork the Almighty]

Perhaps somebody smart enough could start a business defeating Googles filters...

There are plenty of businesses which try and do this. I recall SearchKing getting into a bit of a flap with Google, when a bunch of their customers' sites were downgraded. There are probably thousands of little piss-pot companies that will link to your page a million times for a hundred bucks or some such garbage. People should recognize this for what it is: web-spam. Millions of fake, garbage sites set up for the sole purpose of vastly decreasing the signal to noise ratio, until you can't help but see at least a bit of their trash, at which point they make a dollar.

This is good news

[LoRider]

It's great that Google continues to tweak there search engine to produce the most accurate results. The people complaining are the commercial businesses that are relying on Google search results for free advertising. First off, these "businesses" should not be relying solely on Google search result hits for traffic to their sites, they have to advertising somewhere. Secondly, Google has every right, and duty, to continue battling against businesses from gaming the search results.

My sites have probably increased in position on Google because of these changes, but I don't plan on reducing my advertising budget.

Merchant's Perspective

[sampson7]

Obviously, depending on Google's free side to support a business is risky -- but the alternatives just aren't all that pleasant.

I help run a small business on the side -- http://www.beadstore.com. We are one of the good stores on the web -- we sell a specialty high quality product that has a small but dedicated global following. We provide a fair amount of information about each product and excellent customer service.

We are also reliant upon Google for a huge percentage of our online business.

Realizing that free listings were completely hit or miss, we began advertising on Google through Ad Words. Since then we've been spending anywhere between $500 and $3000 a MONTH on advertising -- which for a small business like ours is a huge hit.

And yes, we've spent countless hours trying to optimize our page position in Google. But we've never resorted to the kind of sleazy tactics some use. So when Google "tweaks" its code and the whole Google Dance shifts, it can mean major rewards or huge costs. What is a small business supposed to do? We have basically two options -- forgo advertising and be forever subject to Google's whims, or pay what amounts to our single largest operating expense (besides inventory) to guarantee some sort of Google traffic. I guess the third alternative is to bypass Google directly, but that's a tough way to go, though we try whenever we can.

So I have a fair amount of sympathy for the good eggs out there trying to run a website business, not because they ever have a realistic hope of becoming dot-com millionaires, but because they love their product. Google is a fickle mistress -- and while we may not have many options, showing a little sympathy when one of us gets dumped isn't really such a bad thing.

One Search Does Not Fit All...

[Anm]

It seems to me that the real source of the problem is in an assumption by Google, and most other search engines, that they can provide a purely string based search without any semantic context.

For the issue at hand, the new filtered results implicitly assume no site can legitimately grab too many links above some threshold for non-trademarked words. But in the case of a shelving provider being referenced as "shelving" (as exampled in the second article), that is not the case. The result is commercial entities with a high PageRank are filtered out.

This is fine for users looking up info about shelving, but not for user looking to buy shelving. Hence my comment on semantic context. In this case, a simple drop down of search prefixes ("I want info about...", "I want to buy...", ...) that select the a particular customized search algorithm in the Google engine. This proviudes enough info to direct the user to a customized algorithm tailored for the users expected type of results. In some respects, this is what Froogle and the other google specialty searches are about, except through the front page interface. This also provides a legitimate hook into bringing blogs back into the fold without interfering other users by looking for reviews and/or user comments.

And in traditional Google fashion, Google could provide links to possible alternative searches in other semantic contexts, just like they already do with spelling and the like.

The prefix approach is only one possibility. Maybe a sentence parser would be better (if you can convince current users to convert from keyword searches).

MetaRefresh=0

[robogun]

It appears they finally did something about that. Most door pages, particularly in porn, stuffed keywords for Google to see, then send everyone else to the porn site via metarefresh=. It avoided SafeSearch and caused situations such as Junior searching on medieval castles for a book report and getting page after page of results sending him to Victoria's Torture and Bondage Castle (with popups).

Also, pages with too many advertiser-only tricks seem to be down. They are still there, but fewer. I guess this is being sidestepped by setting up apache to giving Google ip's one version of a page, and everyone else the real page.

Clearly They Do

[IBitOBear]

Spammers *DO* know about Bayesian filters. Of late a large cross section of my spam is arriving with a random block of words in the semi-non-display tags and footer-esque parts of the page. These words have nothing even remotely to do with the nature of the advertised "product" or "service".

This is clearly a response to the Bayesian filter.

The same hacks who make the spam generating software are right there ready to sell their meta-crime to the web-varnishers.