Slashdot Mirror

← Back to Stories (view on slashdot.org)

Millions of Pages Google Hijacked using ODP Feed

Posted by CmdrTaco on from the well-this-isn't-going-well dept.

The Real Nick W writes "Threadwatch reports that millions of pages are being Google Hijacked using the 302 redirect exploit and the ODP's RDF dump. The problem has been around for a couple of years and is just recently starting to make major headlines. By using the Open Directory's data dump of around 4 million sites, and 302'ing each of those sites, the havoc being wreaked on the Google database could have catastrophic effects for both Google and the websites involved."

47 of 427 comments (clear)

  1. Ugh. This is so not true. by GoogleGuy · · Score: 2, Informative

    This is a placeholder. I'll include more details of why you shouldn't listen to Threadwatch.org in a bit, and debunk this some. Let me get this posted and I'll follow up.

    (Yes, I am GoogleGuy.)

    1. Re:Ugh. This is so not true. by GoogleGuy · · Score: 5, Informative

      Okay, I'll talk about this whole "millions of webpages hijacked! Film at 11!" piece of scaremongering. If you RTFA, the author (and the submitter of the story?) claims that some scraper sites have pulled down a copy of the dmoz RDF, gotten the urls, and are doing 302 redirects to sites in an attempt to hijack them. Note that this does not mean that lots of pages were hijacked at all.

      Here's the skinny on "302 hijacking" from my point of view, and why you pretty much only hear about it on search engine optimizer sites and webmaster forums. When you see two copies of a url or site (or you see redirects from one site to another), you have to choose a canonical url. There are lots of ways to make that choice, but it often boils down to wanting to choose the url with the most reputation. PageRank is a pretty good proxy for reputation, and incorporating PageRank into the decision for the canonical url helps to choose the right url.

      A lot of sites that try to spam search engine indices get caught, and their PageRank goes lower and lower as their reputation suffers. We do a very good job of picking canonical urls for normal sites; sites with their PageRank going toward zero are more likely to have a different canonical url picked, though, and to a webmaster I understand that it can look like "hijacking" even though the base cause is usually your reputation declining. For a long time, it was hard to get anyone to report canonicalization problems, because the site that got "hijacked" would be free-cheap-texas-holdem-plus-viagra-and-payday-loa ns-as-well.com type sites. In fact, I had to offer to ignore the spamminess of any reported sites in order to get people to send in any real data.

      But even though I suspected that this issue affected very few sites, we still wanted to collect feedback to see how big of a problem it was, and to see if we could improve our url canonicalization. So starting a while ago, we offered a way to report "302 hijacking" to Google; I mentioned the method on several webmaster forums. You contact user support and use the keyword "canonicalpage" in your report. Then I created a little mailing list with some engineers on it, and user support passes on emails that meet the criteria to the mailing list.

      So how much reports has all this work (including posting multiple times on lots of webmaster boards to request data) gotten me? The last time I checked, it was under 30. Not a million pages. Not even a hundred reports. Under 30. Don't get me wrong, we're still looking at how we can do better: one engineer proposed a way that might help these sites, and he's got a testset of sites that would be affected by changes in how we canonicalized urls. A few of us have been looking through it to see if we can improve things, but please know that this is not a wildfire issue that will result in the web melting down.

      As a side note, I'm getting a little tired of debunking the source of this story (NickW at threadwatch). For example, he claimed that Google had removed Greg Duffy from Google's index. When I pointed out that he was making an assertion of fact without evidence, he started out revising the story by sprinkling in words like "appears" and eventually pulled the story at http://www.threadwatch.org/node/1822 off his front page. But given that this is the third link to NickW's site from Slashdot in the last couple weeks, I'm guessing that he's tasted the Slashdot effect and wants more.

    2. Re:Ugh. This is so not true. by Anonymous Coward · · Score: 5, Informative

      But even though I suspected that this issue affected very few sites, we still wanted to collect feedback to see how big of a problem it was, and to see if we could improve our url canonicalization. So starting a while ago, we offered a way to report "302 hijacking" to Google; I mentioned the method on several webmaster forums. You contact user support and use the keyword "canonicalpage" in your report.

      I'm sorry, but this is a flat-out lie. If you are the GoogleGuy, then there were 1000+ post threads on WebmasterWorld where people were begging you for input, and you essentially disappeared. I think I might remember seeing one post from you about this "canonicalurl" on a short, almost unrelated thread. You certainly didn't make it clear where to send problem reports, at least not on any of the threads that people were actually reading.

      The fact is, this is a huge problem, and has totally fucked a lot of legitimate site rankings. I honestly believe Google was doing everything in their power to ignore the problem up until now, hoping that it was just a figment of people's imagination, or worse, that it would help increase advertising revenue. And now that it's turning out to be a PR disaster for you, you're in damage control mode.

      I run one of the sites that was affected by the 302 bug. I sent a message to Google about it, and got a canned response essentially telling me there was nothing wrong. I read through no less than 10 threads on WebmasterWorld about this, many with hundreds or even thousands of posts. I saw maybe, maybe, two or three from GoogleGuy. Where were you? Did you somehow miss those threads that spanned 80+ pages??? Why weren't you posting on those threads about this "canonicalurl" thing.

      Luckily there was only one site 302-ing me, and they were doing it by accident and were happy to remove me from their directory. Now I'm back up at the top of the rankings. But I know it's going to be nowhere near as easy for many of the thousands of people who are still affected by this.

      Seriously, that you would come on here and try to discredit someone for bringing attention to a very big problem with Google is pretty distasteful. To me it indicates either a cover-up or having your head buried firmly in the sand. Either way, it doesn't bode well for the future of Google. Instead of flaming people now that the problem is getting mainstream press, why not try and actually fix things.

    3. Re:Ugh. This is so not true. by GoogleGuy · · Score: 4, Informative

      One example is http://www.doi.org/, because people want to have a persistent url like dx.doi.org/10.1226/1588290972, but then be able to have that url do a 302 redirect to a destination page like http://doi.contentdirections.com/mr/humana.jsp?doi =10.1226/1588290972 The destination urls might change, so it's handy to have a persistent digital object identifier on doi.org.

  2. Re:A Real Question by justforaday · · Score: 2, Informative

    According to the previous article (posted a few days ago, and linked to in TFS), a page utilizing this redirect exploit essentially supplants the original page in Google's pagerank listings...

    --
    I'll turn into a supernova and burn up everything. Well I'll turn into a black little hole and you'll turn into string.
  3. 302 by auralrothko · · Score: 5, Informative

    I wasn't sure what a 302 hijack was, so here's the obligatory lowdown for those who didn't rtfa (from article linked page) This exploit allows any webmaster to have his own "virtual pages" rank for terms that pages belonging to another webmaster used to rank for. Successfully employed, this technique will allow the offending webmaster ("the hijacker") to displace the pages of the "target" in the Search Engine Results Pages ("SERPS"), and hence (a) cause search engine traffic to the target website to vanish, and/or (b) further redirect traffic to any other page of choice.

    --
    arg
    1. Re:302 by SassyDave · · Score: 5, Informative
      For the full details of the exploit, TFA gives a pretty decent recipe:
      The technical part: How it is done
      Here is the full recipe with every step outlined. It's extremely simplified to benefit non-tech readers, and hence not 100% accurate in the finer details, but even though I really have tried to keep it simple you may want to read it twice:

      1. Googlebot (the "web spider" that Google uses to harvest pages) visits a page with a redirect script. In this example it is a link that redirects to another page using a click tracker script, but it need not be so. That page is the "hijacking" page, or "offending" page.

      2. This click tracker script issues a server response code "302 Found" when the link is clicked. This response code is the important part; it does not need to be caused by a click tracker script. Most webmaster tools use this response code per default, as it is standard in both ASP and PHP.

      3. Googlebot indexes the content and makes a list of the links on the hijacker page (including one or more links that are really a redirect script)

      4. All the links on the hijacker page are sent to a database for storage until another Googlebot is ready to spider them. At this point the connection breaks between your site and the hijacker page, so you (as webmaster) can do nothing about the following:

      5. Some other Googlebot tries one of these links - this one happens to be the redirect script (Google has thousands of spiders, all are called "Googlebot")

      6. It receives a "302 Found" status code and goes "yummy, here's a nice new page for me"

      7. It then receives a "Location: www.your-domain.tld" header and hurries to your page to get the content.

      8. It heads straight to your page without telling your server on what page it found the link it used to get there (as, obviously, it doesn't know - another Googlebot fetched it)

      9. It has the URL of the redirect script (which is the link it was given, not the page that link was on), so now it indexes your content as belonging to that URL.

      10. It deliberately chooses to keep the redirect URL, as the redirect script has just told it that the new location (That is: The target URL, or your web page) is just a temporary location for the content. That's what 302 means: Temporary location for content.

      11. Bingo, a brand new page is created (never mind that it does not exist IRL, to Googlebot it does)

      12. Some other Googlebot finds your page at your right URL and indexes it.

      13. When both pages arrive at the reception of the "index" they are spotted by the "duplicate filter" as it is discovered that they are identical.

      14. The "duplicate filter" doesn't know that one of these pages is not a page but just a link (to a script). It has two URLs and identical content, so this is a piece of cake: Let the best page win. The other disappears.

      15. Optional: For mischievous webmasters only: For any other visitor than "Googlebot", make the redirect script point to any other page free of choice.
    2. Re:302 by windowpain · · Score: 1, Informative

      Thanks. Both the /. article and the linked story were utterly uninformative. Sometimes it seems that a lot techies disdain even the merest explanation as baby talk. Even when you're addressing a largely technical audience a little explanation helps because not everybody knows every technical detail about an entire field.

      --
      Insert witty sig here.
    3. Re:302 by StrongAxe · · Score: 5, Informative

      I'm still not seeing any explanation of how it works, only what happens when it does work. 1. Phisher creates (say) cïtïcorp.com and makes the home page redirect to the real citicorp.com page. 2. Googlebot browses cïtïcorp.com and gets a redirect to the real citicorp.com, and indexes its contents 3. User does a Google search looking for Citicorp, and finds cïtïcorp.com page that appears to contain the valid data (and it might be the only such page, if the legitimate page gets removed through the duplicate-removal process) 4. User clicks through to cïtïcorp.com expecting to see the valid web page 5. Phisher's server sees that the request is not from a Googlebot, so it serves up a fake page rather than redirecting to the legitimate real one. 6. User believes he is at the real citicorp.com web site, when he is in fact at the bogus cïtïcorp.com website, legitimized by Google. 7. Identity theft. 8. Profit. (OB. Slashdot joke.)

    4. Re:302 by Anonymous Coward · · Score: 2, Informative

      This is where the concept of the 302 comes in. 302 means "I'm redirecting you to a temporary home of crappywebsite.com, located at cnn.com. However, this is subject to change, so you should continue to use crappywebsite.com to find the news content you were looking for". This is, of course, horseshit, but the bot believes it. The bot then happily goes on to index all the information on cnn.com, and links it to crappywebsite.com (the "permanent" URL according to the 302).

  4. Re:Robot.txt by wizbit · · Score: 5, Informative

    No, it means Google has indexed a page that appears (to googlebot) to contain something legitimate, and visiting the actual page by clicking the link silently redirects you to an illegitimate site (usually phish/scam copy of same, etc).

  5. Re:Robot.txt by pluggo · · Score: 5, Informative

    There was an article a little while back on /. that talked about this exploit.

    Site A can return a 302 HTTP redirect to site B when Googlebot crawls their site. The googlebot will then index site B as site A. Site A could have no affiliation whatsoever with Site B; people could be clicking on SesameStreet.com and get AsianHookers.com, etc.

    I do think the figure of millions of pages being hijacked is a little steep, though.

    --
    Pulling together is the aim of despotism and tyranny. Free men pull in all kinds of directions. It's the only way to mak
  6. How to check if your site is being hijacked... by ites · · Score: 4, Informative

    1. search Google for 'allinurl:', e.g. 'allinurl:slashdot.org'.

    2. copy and paste any dubious URLS into this tool and check whether they're using 302 redirects or not.

    3. Panic! /me notices that my company's web site has been thusly hijacked... and yes! Doing a Google search on the main text on my company's web site shows dozens of unrelated sites high in the ranking. None of these actually have the text on their pages.

    One example: http://www.tradedoubler.it.

    Luckily, the phrase in question is complete gibberish and no-one ever finds our site through Google, only by reputation and word of mouth.

    Still, I think it's clear Google have a serious problem here...

    --
    Sig for sale or rent. One previous user. Inquire within.
  7. Good explanation about 302 hijacking by angio · · Score: 4, Informative
    Someone posted a nice explanation of the phenomenon at webmasterworld.com.

    302 hijacks work because Google goes to http://bad.site/ and gets redirected to http://good.site/. It then treats the contents of the bad.site as identical to that of good.site. The effect seems similar to if somebody simply copied an entire page off of your site (I'm not sure if it's actually more serious than this), but it's easier to do because you're just keeping a small table of redirections.

    How serious is it? Don't know. It's pretty easy for a webmaster to check for hijacking and have her pages de-hijacked (see aforementioned article). It's probably not as screamingly awful as the threadwatch.org article suggests, but the redirector sites are rather annoying. Several of the comments in the webmaster article suggest that Google has already started moving on the problem.

    1. Re:Good explanation about 302 hijacking by Col.+Klink+(retired) · · Score: 2, Informative

      > The effect seems similar to if somebody simply copied an entire page off of your site (I'm not sure if it's actually more serious than this), but it's easier to do because you're just keeping a small table of redirections.

      The key here is that only googlebot is redirected. If you simply copied someone else's site, everyone would still get the info they were looking for. However, if you only redirect googlebot, you can redirect others to whatever you want.

      --

      -- Don't Tase me, bro!

    2. Re:Good explanation about 302 hijacking by Anonymous Coward · · Score: 1, Informative

      You can also serve different content to googlebot if you copy the whole page, so there's really not much of a difference. It's called cloaking: people have been doing that for ages.

    3. Re:Good explanation about 302 hijacking by squiggleslash · · Score: 4, Informative
      It's a little more than that. It's not just that bad.site is treated as identical to good.site, it's that good.site is potentially removed from Google. "302" means "temporary redirect", which gives Google the false idea that good.site isn't a permanent website.

      Whether it actually removes good.site from the index has to do with, apparently, the PageRank of both sites.

      It really wasn't until I read a full explanation and they covered that bit that the whole thing clicked for me.

      --
      You are not alone. This is not normal. None of this is normal.
  8. Re:I've had it with Google! by 33degrees · · Score: 2, Informative

    I know you were trying to make a joke, but if you'd RTFA you would know that MSN is as susceptible to this as Google is. Only Yahoo has addressed the issue.

  9. Wrong by PornMaster · · Score: 4, Informative

    301 is a permanent redirect, 302 temporary.

    This is why the "302 hack" works. If the redirect is only supposed to be temporary, the search engine keeps the URL of the 302 as the URL for the document, but indexes the content of the page to which the redirect is directed.

    301 is what you should be using to point the SEs to your new pages if you've moved them. The behavior is supposed to be for the SEs to replace the old URL in their index with the new one, and furthermore count all links to the 301ed URL as being towards the new one. I don't know why it's not working for the grandparent poster, but it's the way that the functionality is "advertised" for Google and Yahoo, and it should work.

    --
    500GB of disk, 5TB of transfer, $5.95/mo
  10. From the Google "Information for Webmasters" by YouMakeMeSoANGRY · · Score: 5, Informative
    Google claim...
    Fiction:A competitor can ruin a site's ranking somehow or have another site removed from Google's index.
    Fact:There is almost nothing a competitor can do to harm your ranking or have your site removed from our index. Your rank and your inclusion are dependent on factors under your control as a webmaster, including content choices and site design.

    How about adding "Fiction: Google information for webmasters contains any facts"?
  11. Re:Robot.txt by AssHatAnonymous · · Score: 3, Informative
    No, it's the other way around. Someone has access to asianhookers.com and redirects to sesamestreet.com. When googlebot then correlates asianhookers.com with sesamestreet.com and depending on some unknown formula decides which domain is the actual owner of the page. So that if the formula decides that asianhookers.com "owns" the pages on sesamestreet.com (because of the redirect) then when google is building their links they print, in the text of the page, sesamestreet.com, but in the html of the page they actually link to asianhookers.com.

    It's fucked up.

  12. Re:Google is borked by Anonymous Coward · · Score: 1, Informative

    So try Teoma instead. They're not as well known as Google but I find they return much more relevant results in many cases.

  13. pure FUD the submitter is a spammer by Anonymous Coward · · Score: 4, Informative


    what major headlines ? millions of pages !! the world is coming to an end !!!!

    a quick whois on threadwatch.org (the submitters site) reveals its hosted by search engine spammers
    platinax.co.uk which is registed to a UK "company" called BriteCorp
    http://www.britecorp.co.uk/

    who offer all the usual SE spamming methods
    coincidence ?
    a whois on britecorp's platinex site reveals they have removed their address from the whois db, and their websites contact details are a mobile phone number (07963 808470)
    further investigation on britecorp reveals they are not a "real" company but trading as "Brian Turner" (pic) and companies house dont seem to have any records of any of these companies, though iam sure further investigation could find out more

    so why would a supposedly reputable marketing company have a cell phone as a primary contact point ?
    something to hide egh ?
    or perhaps local trading standards would like to hear about them and their "services" ?

    northern scum by any other name

  14. Re:Robot.txt by arkanes · · Score: 5, Informative
    One problem is that people use 302s when they should be using 301s, like directory sites. No doubt this is because they want to get referral counts up.

    A 302 is a "temporary redirect". Basically, it says that the content normally lives at the URL you requested but that, just this once, you should look at this other URL for the content. Googles response to a 302 is actually very reasonable. I suppose the best thing they could do is just not follow 302s.

    A 301 is a permanent redirect, indicating that the page isn't at the original URL and that all future requests should be made to the new one. I don't know what Googlebot does in this case but I assume it discards the original URL, which is what the standard recommends.

  15. Re:But what's the point? by micromoog · · Score: 3, Informative

    The hijacker's script watches to see who's coming. If it's googlebot, redirect. If it's an actual user, do [insidious thing].

  16. Kindly extract your head from wherever it is by ites · · Score: 5, Informative

    This story does not need "debunking".

    What it needs is a rapid and satisfactory answer or Google will find themselves at the receiving end of more angst than they even know is possible.

    A concrete example. My company's web site has been in existence since 1995. So we have pretty good page ranking. Our main page has one phrase, very distinct, unique.

    When I search for this phrase (in quotes), Google reports hundreds of matches. These sites (except our own) do not contain the phrase but are sites that sell traffic boosting.

    The 302 problem is real.

    Incidentally, I just spent 15 minutes at Google.com looking for a way to report the problem. Where is that mention of "canonicalpage"? In the bottom shelf of a filing cabinet, behind a locked door that says "beware of the tiger"?

    I'm not surprised you got only 30 reports. What I am surprised at is that you appear to speak for Google yet have such an inane response to what is a real (and for many people, a terrifying) problem.

    --
    Sig for sale or rent. One previous user. Inquire within.
    1. Re:Kindly extract your head from wherever it is by Anonymous Coward · · Score: 0, Informative
      Incidentally, I just spent 15 minutes at Google.com looking for a way to report the problem. Where is that mention of "canonicalpage"? In the bottom shelf of a filing cabinet, behind a locked door that says "beware of the tiger"?

      Obviously you did not google for it. Try this

    2. Re:Kindly extract your head from wherever it is by alphakappa · · Score: 2, Informative

      Here's where you can file a report.

      --
      "When the only tool you own is a hammer, every problem begins to resemble a nail." - Abraham Maslow (1908-1970)
    3. Re:Kindly extract your head from wherever it is by infernalproteus · · Score: 2, Informative

      "beware of the tiger"

      You mean "beware of the leopard"...

  17. Re:Robot.txt by PornMaster · · Score: 4, Informative

    A million may be no harder than four to hijack, but a million dummy sites that would actually fool people is much harder than four.

    This isn't about fooling people, it's about fooling a flawed technology to get false listings in the search engine results pages. It's about getting a lot of traffic. Yes, some people will be really pissed off when they get redirected to an affiliate program or something of the sort, but some small percentage of people will buy. If the cost to bring in a million visitors is miniscule because you're stealing search engine placement, and you get 50 people to sign up to something that pays you $50 a person, then you're up $2500 minus your hosting costs.

    $2500 to someone in Malaysia is a lot of dough for a little coding... they could work for $200/mo in some kind of outsourcing plan or make a year's wages in their spare time. What do you think they're going to do?

    --
    500GB of disk, 5TB of transfer, $5.95/mo
  18. No, it's not about redirecting the user... by ites · · Score: 5, Informative

    It's about pushing unrelated sites up in the rankings.

    For instance: I have a site with excellent page ranking. Now a new site will set up, and do a 302 to my site. Google now gives this new site my page ranking. When the new site is indexed, it removes the 302 redirection.

    When you search for my site, you now find these new sites instead. There is no redirection when you click on a link, the the "cached text" that Google shows is wrong.

    Basically this technique allows people to get high page rankings without earning them. It's very widespread - I counted over 60 such parasites for my company's web site (which has excellent page ranking).

    --
    Sig for sale or rent. One previous user. Inquire within.
  19. You got an email from me! by pastepotpete · · Score: 3, Informative

    And I know two other people who sent one. Maybe you should check again? I doubt me and my mates account for 10% of your responses. If you believe that the people affected by this are all "spammers" then perhaps the problem is false positives for your spam detection filters. In fact you should probably take a look at your spam detection filters anyway. Last time I checked--probably much more recently than you checked for canonicalpage emails, there was a bunch of scraper sites running AdSense where good relevant results used to be.

  20. Why This is Such a Big Deal (A Summary) by Anonymous Coward · · Score: 5, Informative

    This was originally posted the first time a story about this ran, but since a lot of people are still confused, here it is again...

    There seems to be a lot of confusion as to why exactly this is such a big deal. A lot of people saying there's no problem or that this is nothing new... basically just not understanding the issue. Let me explain:

    Suppose you have a small business under the domain http://xyz.com/, and search engines bring you a lot of traffic because you rank high for keywords in your market. You have a lot of people out there linking to you, a lot of satisfied customers, good content on your site. You're always in the top 10 somewhere when people search for "xyz widgets".

    Well, this issue with Google makes it very easy -- incredibly easy -- for someone to knock your site out of the rankings entirely. And I mean for *everything*, to where searching for your own company name in quotes literally buries you hundreds of pages deep in the results. We're talking sites going from getting 1000 unique hits to 10 overnight.

    And here's the kicker: It requires absolutely no technical knowledge, no time investment, and is perfectly legal...

    All I have to do is have another domain handy that is roughly as popular as yours. And I make a "links" page, like one of those directory services, that lists your website. But instead of being a normal hyperlink, it's a CGI (or PHP or ASP or whatever) script that generates a 302 redirect to your domain... Now, these are very simple, common scripts. One-liners that you can download from cgiscripts.com and stick on your server. The original intent of these scripts is to track which links are being clicked on your site. But now they've found a new use, because when Google gets that 302, all hell breaks loose.

    See, according to the HTTP spec, 302 is a *temporary* redirect, which means Google is supposed to interpret whatever content it finds at the 302 target (your site) as really belonging to the URL of the source (my site). Google is just obeying the spec strictly here, and with devestating results. Why? BECAUSE THE DUPE FILTER NOW KICKS IN! You see, Google has a "dupe filter" that says if the same exact content is found for two unique URLs, then one of the URLs is obliterated in the rankings. Because after all, searchers don't want to be finding the same content over and over. If that happens, they'll start using a different search engine. But Google, sticking strictly to the HTTP spec, doesn't know who the content really belongs to when it gets a 302.

    So Google essentially flips a coin. And if it comes up tails, say bye-bye to your domain in the rankings. Your *entire* domain. Because the dupe filter isn't limited to just the page that the 302 is pointing to -- it applies across your entire domain.

    These 302 "exit-link-trackers" are all over the web. They've been used by webmasters for years. But it's just recently that Google has started treating 302 this way, so it didn't have any bad effect before. But now it kills you.

    The funny thing is, the solution seems pretty simple: Just stop treating 302s this way if they point to a different domain. But for whatever reason Google isn't listening. Hopefully the press that's being generated now will give them the kick in the ass that they need.

  21. Re:I've had it with Google! by Waffle+Iron · · Score: 2, Informative
    MSN is as susceptible to this as Google is.

    That's only because Microsoft dropped the original vision of MSN, which was a closed centrally controlled service like a glorified BBS. When it was introduced, they planned to leverage their desktop dominance to get the entire world to subscribe to their proprietary network.

    The original MSN user interface that was bundled with Windows 95 looked more like the Windows file manager than a browser. I imagine that if MSN had continued down that path, then searching for information would today look more like some versions of the MSDN library help browser (based on a manually controlled central index) than like Google.

    As it turned out, people preferred the freedom offered by the real Internet, and their plans never panned out.

  22. Re-re-explained by fizbin · · Score: 5, Informative

    Okay, so basically this is the problem: when Google encounters a status 302 redirection (as opposed to the status 301 redirection) it then indexes the content as belonging to the initial URL, not the URL at the end result of the 302 redirection. Other things happen later because of google's design.

    302 redirections are temporary redirections - the idea is that a 302 is supposed to be used when someone needs to be redirected to a new page, but should still use the original URL if they want to come back later. As an example, the page http://purl.oclc.org/OCLC/PURL/CONTRIBUTORS performs a 302 redirect to http://purl.oclc.org/docs/contributors.html. This means that although your web browser needs to go to some other URL for the content at the moment, they really should remember the first url as the permanent one.

    Contrast this with what happens when your browser visits http://snowplow.org/martin - you get sent a 301 redirect to http://snowplow.org/martin/. (Note the extra slash) In this case, the server is saying "the url with the slash on the end is the real location, and you should not try to come back here without the final slash in the future."

    Ideally, if every web browser behaved according to spec., bookmarks (remember bookmarks?) would get automatically updated to the new URL when you selected them and the redirect was a 301 redirect. However, for a 302 redirect, the bookmark would stay as is.

    302 redirects can be very useful when you want to set up a hierarchy of "logical" URLs that will permanently point to the correct location. 301 redirects are useful when you're obsoleting an old URL and wish people to go and use the new URL from now on.

    Okay, so how does this relate to google? Well, let's suppose that you have a great site on fruitbats. I can set up http://www.example.com/topics/fruitbats to be a 302-style redirect to your site, essentially saying "The information at http://www.example.com/topics/fruitbats is temporarily being hosted by http://www.yoursite.com/". Now, google when it spiders pages will see that, will go retrieve the text from your page and will then index it under http://www.example.com/topics/fruitbat, since after all I just gave a temporary (302) redirect.

    But it gets worse, because a final part of google's indexing process is to compare pages for identical text, and throw out all but one of the URLs. Apparently this stage has nothing to go on other than the text and the recorded URLs, and so your URL stands a fifty-fifty chance of being thrown out.

    Except that I've not just redirected http://www.example.com/topics/fruitbats to your site, but also http://www.example.com/topics/fruitbat, http://www.example.com/topics/fruit_bat, and http://www.example.com/topics/fruit_bats. Now your lone URL doesn't stand much of a chance of being the one kept by the "throw out duplicates" processor, does it?

    In a sense, of course, there's little google can do to prevent this, because even if they weighted 302-redirects lower in their "throw out duplicates" stage, I could always just go snag a copy of your website each time googlebot visits, in essence doing the redirection myself. (How? Just search the apache mod_rewrite guide for "Dynamic Mirror") However, doing it through 302 redircts means that google pays for the bandwidth to go get your page, not me. (Not that this is necessarily a signficant amount of bandwidth, since we're only talking about basic google here and not images. Depending on the revenue you get by misdirecting google queries it might be economical)

    Of course, for this to really work, I'd need a list of websites sorted by category to build up my redirect db. But wait! The ODP feed provides exactly that.

    I am a little bit wary of doi

  23. Re:RTFA by mla_anderson · · Score: 5, Informative

    No, the way it works is with the 302, but only for the googlebot.

    1. Googlebot goes to scammer's site
    2. Googlebot is given a 302 (redirect) to the victim's site
    3. Googlebot indexes the victim's site as belonging to the original URL
    4. Googlebot goes to the victim's site
    5. Googlebot realizes this URL is already indexed and "belongs" (according to the Google code) to the scammer.
    6. The victim's site get's lower rankings as the page is not even indexed, the scammer's site gets a higher ranking.

    For this to work the scammer has to give the 302 only to the googlebot, all other browsers need to get the content of the scammer's page. If you google for "cheapest car insurance" (IIRC) you can find an example of this. Change your User Agent accordingly and click on the top Google link, you'll end up at another site. Change back to Mozilla and you'll get the scammer's site.

    --
    Sig is on vacation
  24. OK, an example by ites · · Score: 2, Informative

    My company's web site is imatix.com

    You will notice that the site's main page contains very little text. There is one marketroid phrase, "Strategic solutions for a complex world".

    Now search Google for this phrase.

    Look at the results. A completely irrelevant site has come in at first place. imatix.com is now at second place (this changed today).

    imatix.com is an old site, with very high page rank. Now, it does not matter much for us, since no-one is going to search for this phrase, but if this can hit imatix.com, it can hit other sites.

    The problem is entirely real, and it is extremely serious. I'd say, if Google don't fix this before it hits the main media, they will suffer irreparable damage to their reputation.

    --
    Sig for sale or rent. One previous user. Inquire within.
    1. Re:OK, an example by GoogleGuy · · Score: 4, Informative

      Thanks for the concrete example. As someone else pointed out:
      - for the search imatix I see you at number one.
      - for the search "Strategic solutions for a complex world" I see you at number one.
      - for the search allinurl:imatix.com, that search (and it's sister operator inurl:) only look for the words in the url. So it's perfectly fine to show results like "real-imatix.com/" because they contain the word imatix. These results are not hijacking results--this is expected behavior for inurl and allinurl.

      Hope this helps,
      GoogleGuy

    2. Re:OK, an example by GoogleGuy · · Score: 3, Informative

      Just to follow-up, I saw your email come through the queue from user support. The engineer who checked it out basically said "They appear at the top of the results when I do a search. Still, just because their website only has that one phrase on it doesn't guarantee that their site will appear at the top of the results." So this isn't a "302 hijacking," but I hope our user support will reply in addition to my post. :)

  25. Re:Robot.txt by northcat · · Score: 4, Informative

    This is more like one site hijacking the ranking of another site. Suppose you're Ferrari and I'm the hijacker. You have ferrari.com and I have irule.com. Since you're ferrari.com you get very high rankings when people search for "ferrari" on Google. You're probably the first site displayed. And in the results page on Google, it displays a summary probably like "the official home page of ferrari cars". On my website I set up a 302 redirect to your website. It means, when someone visits my irule.com, they get redirected to ferrari.com. I don't do anything to your website, I don't have access to your website. I hope you know that Google indexes web pages by visiting those webpages with the user agent string "googlebot" and, of course, Google's IPs which are known to people. When Google sees that my page is 302 redirecting to ferrari.com, for certain reasons, it replaces ferrari.com in its index with irule.com. So when someone searches for "ferrari" the get irule.com as the first result instead of ferrari.com, and the summary still says "the official home page of ferrari cars". Now, I only 302 redirect irule.com to ferrari.com when googlebot visits my page. When anyone else visits irule.com, I give them something else, probably lots of ads, or I redirect them to some other site like LotsOfSmut.com. So I'm "hijacking" any references to ferrari.com on Google and its ranking. And when someone searches for "cars", instead of ferrari.com as the ninth result, irule.com is displayed. So... I profit (you do the math).

    (Sorry for dumbing down my post so much, too much experience explaining things to my grand mother)

  26. Re:Robot.txt by ReverendLoki · · Score: 4, Informative
    The key is that they are using a 302 redirect, which is used to signify that the redirect is temporary only. In a completely honest and trustworthy Internet, this is used to indicate that for whatever reason (HW failure, slashdotting, etc), the requested pages were temporarily unavailable on the main site and were being hosted elsewhere until the issue can be resolved. This is telling Google et al that the content being redirected to (Sesame Street, for example) is normally hosted on the redirecting site (Asianhookers). From then on, whenever Google returns the result of the Sesame Street pae, it is listed with the URL pointing to the Asianhookers page. It does this under the assumption that once the issue requiring the redirect is resolved, people will want to go to the "original" page, and will still be redirected to the content in the meantime.

    Aside from a filter on Google's end to resolve this, it would be nice if the practice of using 302 redirects also included a means of confirmation of the setup on the site being redirected to. If the site actually hosting the data does not in some way confirm the redirection, either through a tag in the header of the html, or perhaps in a third, predictably place file (much like a robots.txt file). Of course, this would first require te standard to be rewritten, and then would require people to actually abide by it.

    --
    09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
  27. Re:My site is affected by GoogleGuy · · Score: 4, Informative

    Yeah, this is a common misconception. allinurl: and its sister operator inurl: look for terms matching in the url. For a search like [allinurl:thehumorarchives.com], a result like www.stumbleupon.com/url/www.thehumorarchives.com/f orums/ is a fine result, and doesn't have anything to do with this.

  28. Re:HTTP 301 filter by accidentalGeek · · Score: 2, Informative

    Ach! this leads to an endless loop. Please note my revised (and more complicated)version

  29. Re:clsc.net seems to be down... by GoogleGuy · · Score: 2, Informative

    It's me. I've had the GoogleGuy handle since Jan 19th, 2005. From the K5 article, the allinurl: stuff isn't true though; allinurl: just looks for term in the url. So [allinurl:imatix.com] can show results from any site that has imatix in the url.

  30. How and when Yahoo fixed it by clsc · · Score: 2, Informative

    Sorry for not writing this in the article - it's pretty long already and you just have to cut somewhere, but here goes:

    Yahoo was exactly as vulnerable as the rest of the search engines. In fact this problem was pretty bad with Yahoo at one point. What Yahoo did was simply to fix it by implementing some internal rules about how to interpret redirects.

    I believe it was fixed around June 2004 - at that time the problem had already been known (and aboused) for a long time, but use was not widespread yet. The details of the fix can be seen on this one-page PDF

    It's simple (and identical to the solution i suggest in my article): When "Yahoobot" (actually it's called "Slurp") sees a 302 redirect, it checks if the domains of the redirect and the target are the same. If the redirect is from one domain to another, Yahoo keeps the URI from the target domain. If the redirect is from one page to another on the same domain, Yahoo keeps the "source" (ie. the redirect script URI).

  31. Absolute hilarity by brian_turner · · Score: 4, Informative

    Absolutely Roflmao!!

    I guess some people have never heard of the term "sole trader". :)

    My internet business is barely a year old - almost everything is communicated with other webmasters via e-mail - phone support is provided as a last option, but it means that if anyone really needs to use it, then they can have my immediate attention wherever I am, to have their concerns addressed immediately. :)

    As for spamming - well, this is one of those "anonymous cowards" some of us are familiar with, who believes that if you purchase a link from another site, or become involved in a link exchange, or register your site in a directory - then you're a spammer. :)

    Thanks for the heads up on the Platinax registration details, though - hadn't realised they'd been left out. I had a run in with some Belgian Nazis last year, after I booted them from a forum I admin, when they tried to use it for promoting Neo-nazi propaganda. They've tried a few times to get back at me since, so I've been trying to reclaim some privacy online. Platinax reg details should be public, though - I'll put something online, then try and fine a PO Box for the hate crap.

  32. Re:THANK YOU! by GoogleGuy · · Score: 2, Informative

    You bet. If you want to make sure that we have the info to check it out, you can go to google.com/support and when you get to a form where you can enter info, just use canonicalpage as the subject line. We are collecting data to user support to build up a testset for checking any changes we want to try.