Slashdot Mirror
Google Blocks 'Optimized' Pages
Rhett Creighton writes "For the past few years, webmasters have found tricks that bring their page higher for a given keyphrase search. Google recently implemented a filter to block sites that appeared to be tricking it into gaining a higher ranking. This NYTimes article reports of angry retailers who are losing their businesses, while this article gives more technical conspiracy theories of what google is actually doing."
With the huge number of postings on all the various forums, concerning this update, most people don't know where to start looking for information about the recent Google update. The following is an attempt to put down rationally (I hope) most of the information that is known and the (unproven) theories behind the update algo.
Introduction.
Starting on the 16th of November, a major shift in results was seen on Google. Veterans recognised that Google appeared to be doing a major update, not seen for many months, as reported first on WebMasterWorld who named it Florida, continuing the tradition of naming updates rather like hurricanes. In this case it was a hurricane! As was usual with many updates, there were moans and groans as people complained about their sites falling. Many people were unaffected (including us) but the symptoms of the sites being dropped were not usual. No penalties, such as PR0, seem to have been applied against pages that had fallen - yet none of the pages targeted at specific key phrases, particularly index/home pages, appeared in the top results for these search terms. Indeed some had dropped hundreds of places and, in some cases reported, off the scale. Yet these pages did appear for obscure phrases and were obviously still in the index.
It appeared to us and to several other respected names (though hotly disputed by others) that some sort of over-SEOd filter had been applied to check if overt SEO had been done for that particular phrase. It was as if Google were checking to see if external links to the site included the phrase, on-page optimisation was being done for the phrase and even if the domain included the phrase. If the density of the optimisation, both on and off the page, appeared too artificial, then a filter was tripped and down went the page - solely for that phrase.
Google had never looked favourably on abuse of their systems and many established SEOs looked upon this algo tweak as a way of Google getting rid of the abuses of links and stopping the scrambling for getting (and sometimes buying) links including your required anchor text from other high PR, but probably irrelevant to your subject, sites. It seemed to make sense.
On Friday, 21st November, Google decided to tighten the filter. All hell broke loose as tens of thousands of sites disappeared from positions they had held (in some cases) for years. We noticed some of our client sites plummeting for their major key phrase from being #1 to total invisibility. Yet this was only in highly competitive areas, not for their secondary phrases. These sites were, in most cases, not highly optimised, had not sought reciprocal links but had achieved their rankings through being on the web for 4 or 5 years. The bad news was that their company name and domain included the key phrase, sites (including directories) linking to those sites included the key phrase in their links and Google interpreted this as over-optimisation and down they plunged. In many areas all the top 20 ranking sites disappeared, including industry leaders, to be replaced by educational sites, news review sites, government sites, major shopping portals or directories. Something major had happened - but what?
The Facts!
Thousands of web pages have been suddenly demoted in the Google search results, primarily on the main commercial search terms for which they targeted their pages to be replaced by other sites who, in the main, referred to the search term obliquely. Several were the main shopping portals or business directories which gave listings for companies who may provide the services requested, many were not.
Very high-ranking authority sites seemed to be unfiltered.
The changes were starkly obvious on regional English language Googles where a regional filter was employed and there were less commercial sites with authority.
An example for Google UK is the search for the word shelving. On the
Socialism: A feeling of discontent and resentment caused by a desire for the possessions or qualities of another.
If you want just prices and consumer related crap, I suggest google's sister site
froogle
Google has a specialized tool to access their search engine specifically to do shopping/price comparison... So yes, you are missing something. :)
Besides, these sites were using hacks to artificially inflate their pagerank instead of providing a higher quality site to increase it.
Maxim: People cannot follow directions.
Increases in truth directly with the length of time spent explaining them
About 50% of the time when I'm searching, I AM looking for vendors of a product in order to do price comparisons
Google already has a search engine specifically designed for price comparisons... maybe that's what you're missing.
It's called Froogle.
I'm not sure why they haven't added a tab for this on their main page, as it would make a lot of sense to separate out commerce-related searches from information-related searches while making both easily accessible.
Maybe partying will help...
The writeup made it sound as if the NY Times article was about upset retailers mad at google when really they're upset at the company that was tricking the search results.
"Common sense" comes to mind.
If you read the NYT article, it clearly says that some unscrupulous vendors were clogging up the search results. So, on the first page of Google results, you'd get most of the sites from the same vendor (shell sites, put up specifically to increase the number of links between them, thereby increasing the PageRank).
Google is trying to level the playing field, so that no one site can dominate the results.
Looking at your complaint, I think it would make sense for Google to have a "vendors" checkbox, which would list sites selling stuff, as opposed to sites giving out information.
The answer can be found for your problem by
visiting http://goatse.cx/robots.txt
If you don't care for visiting the site ( I wonder why)
you will see they don't want any bots in the site.
i've kinda felt like in the past few weeks i've noticed my searching improved. it seemed for a while that i would never see a homemade page without going to page 6 of results. in addition, even though .net .org and .com aren't followed as well as they were a few years ago, if you do a search that will just include .com sites, you will see more companies turn up
just put "site:com" in the search.
what im worried about right now is google doing the ipo thing and becoming a lot less useful. we will see.
Google is in the business of providing search results, not providing companies that find ways to "trick" the ranking engine into getting free advertising.
Taken directly from google's technology explanation page
-----
Integrity
Google's complex, automated methods make human tampering with our results extremely difficult. And though we do run relevant ads above and next to our results, Google does not sell placement within the results themselves (i.e., no one can buy a higher PageRank). A Google search is an easy, honest and objective way to find high-quality websites with information relevant to your search.
-----
To me that says: if you succeed in tricking the system, don't base the success of your entire business on it because it won't last forever.
I mean seriously, ever try to search for something legitimate, like Infrared Photography and come up with 700 pages related to some kind of XXX web site because someone figured out that if you put an infrared filter on some sony camcorders you can see through womens clothing? Of course, all 700 links point to a site that is totally unrelated to either legitimate Infrared Photography, or seeing through womens clothing. It's very frustrating from a user's perspective (no matter which user you are!).
i don't think you did. if you did, you would have known that they gave a prime example of a larger company fouling up the results and really hurting a small company that had been doing fine before the larger company started spamming google.
... how many people do you expect to actually go to the 2nd page?
case in point, last night, i decided i was going to buy the "i'm a bomb technician. if you see me running, try to catch up" shirt. when i searched for it, the first 7 results were more or less the same page (different bgcolors, different rotating ads, different popups), which all pointed to the same url for checkout. i didn't find the vendor i wanted until the 2nd page
i guess ihbt.
vodka, straight up, thank you!
Absolutely agree. I do a little web site consulting on the side and I usually tell my clients three bits of advice for better exposure:
1. Think about putting some kind of unique, useful, and/or entertaining content on your web site that people will want to visit, link to from their own web site, and even email to their friends. Good content builds traffic.
2. Take basic steps to make your pages search engine friendly. Descriptive titles, simple honest meta-tags, useful text in every page, descriptive links to other pages, etc.
3. Don't be obsessed with your Google ranking. Don't give money to anyone who claims they can boost your Google ranking. If you want to spend money for traffic than buy Google Ad Words or sponsor links, at least that way you pay per actual click-through rather than paying into a bidding war for an uncertain better ranking.
The problem being that the vast majority of people doing a search for 'Dawson Creek' are probably looking for the TV show and not your small town in British Colombia. Don't ask me why, the show is terrible. So surely this is a different problem than search obscuration by comerical web spam? That search obscuration by fans of seriously bad teen drama.
Warning! This post may contain a pun!
A few days ago I tried a seach for "cellular customer satisfaction". The first several pages were bogus resellers (many of them the same page under different URL's). None of them contained the kind of information I needed about how customers rate the various cellular service providers. This morning the same search is yielding lots of useful data instead of the fake spam-like pages I had been getting.
KUDOS to Google for fixing this! Whatever changes they've made to their pagerank algorithm, Google is suddenly working again like I expect.
Life is short: void the warranty.
Google's "PageRank" formula is their top-secret way of determining in which order to display websites for any given keyword. Everyone knows that refering links is the main component of PageRank, but Google has always been hush-hush as to what else is included in the formula.
It's also known that PageRank isn't a static formula. Google reserves the right to change it at any time, in what is known by Google-watchers as a "Google Dance".
The only legit way to be highly ranked by Google is to be the most authoritative source for information about whatever you discuss, and naturally links will form from other quality websites on your topic and up the PageRank scale you go. www.microsoft.com being a 10/10 ranking doesn't indicate that Google likes Microsoft, it just simply indicates that site is the most authoritative site about a topic a lot of people talk about, Microsoft's products.
Any other way to cheat the system will result in penalties applied to your score. It's not so much a filter as it is negative factors in the formula. Google steadfastly claims that it doesn't maintain a blacklist of "bad" sites, but it is clear that sites designed to cheat Google's PageRank formula always fail once Google tweaks the formula. They don't need a blacklist, they simply identify the characteristics that define a "link farm" and then apply a penality. If a given site has a lot of links to external domains, very little non-link content, and absoulutely every linked to site returns a link back to the orignal site, it sure smells like a link farm and that's what the system penalizes.
To put it bluntly, anybody who's business depends on being displayed on the first page of Google results should be buying AdWords placements. If you're working hard to stay #1 in the editorial results, you're never gonna win. And no, just because your business depends on it doesn't mean you get to sue when Google pulls an ill-gotten #1 ranking out from under you.
Nonetheless, I think the gains outweight the losses. Spam results are obnoxious.
I've always been pretty happy with Google's responses as well. They take the time to make a personalized response. I emailed them a question awhile ago about how one would go about finding what sites are linked to a specific site. The responses I received the very next day:
Hi Dave,
Thank you for your note. Yes, we do offer this kind of search. To find the pages that link to any given URL (say www.stanford.edu , for instance) go to the Google advanced search page at http://www.google.com/advanced_search and do a 'link' search. If you do not want to use our advanced page, you can perform a link search directly from the Google search box by typing link:Stanford.edu or link:www.Stanford.edu
This link search, however, may not return a comprehensive set of results. The results show a sample of the links that point to a page, but this list is in no way indicative of the link structure utilized by Google to formulate a page's PageRank.
To obtain a comprehensive list of the links that point to a page, perform a Google search on your URL. From the result page displayed, select the
"Find the web pages that contain the term" link and Google will provide you with the web pages that mention the address.
Regards,
The Google Team
But if you enter just plain "goatse" and hit enter, you'll get a google directory for Scientology =)) See here
yush
Funny -- one of the sites I run recently took a hit in the rankings, and we didn't do much optimization. I researched it last night and found that Google has just rolled back to a very old version of their data. Last week the title tags they were showing for our site were up to date, but now they are showing data from over a month ago.
I wonder if this rollback was in response to the complaints?
In any case, without much trickery we were able to get relatively good ranking in the latest (pre-rollback) version just by having actual content and decent title & meta tags.
Cheers.
If you have trouble finding reviews of peripherals among all the links to stores selling the peripherals, add one of the following keywords to your search:
I was looking for ringtones for my Ericsson phone a few weeks ago, and (literally) the first 3 pages were all sham sites that were filled with ads and lead back to - you guessed it - www.ringingphone.com!
Those sham sites are no longer showing up in the Google results, and I can actually find what I was looking for instead of having to wade through the cesspool shopping mall that the internet is turning into.
Advertisers are the only ones that Google has to answer to. If they do something that makes their advertisers go away, you can bet that they will quickly reverse that decision. And, the only thing that will make advertisers go away is whatever makes end-users go away.
We did a case study about this in one of my MSc classes. The example was Guinness: its end users are obviously drinkers, but it has no direct commercial relationship with them. Rather, it sells via pubs, which are owned by brewers, and Guinness competes with brewers. So, Guinness has to walk a fine line between keeping its end users happy while not pissing off its delivery channel. Ultimately, there is a "sweet spot" where Guinness markets to end users, end users go to pubs to buy Guinness, the brewers and Guinness itself split the profits. But if either Guinness or the brewers who own the pubs gets too competitive, the synergy evaporates and everyone including the end users loses out.
Froogle listings are limited to the US based companies only.
They're working on it. It's called Froogle.
Just FYI
http://slashdot.org/articles/01/03/16/1256226.shtm l (4th post from the top) is why scientology and goatse.cx are related on google.
Read my writeup on it at insaneone.com/13/01