Slashdot Mirror
Millions of Pages Google Hijacked using ODP Feed
The Real Nick W writes "Threadwatch reports that millions of pages are being Google Hijacked using the 302 redirect exploit and the ODP's RDF dump. The problem has been around for a couple of years and is just recently starting to make major headlines. By using the Open Directory's data dump of around 4 million sites, and 302'ing each of those sites, the havoc being wreaked on the Google database could have catastrophic effects for both Google and the websites involved."
Nothing for you to see here. Please move along.
OMG!!! Slashdot's been hijacked too!
I'll turn into a supernova and burn up everything. Well I'll turn into a black little hole and you'll turn into string.
What gains are made when someone hijacks a web site? This has probably been discussed before, but I'm too lazy right now to look it up. Anyone?
"I'm just here to regulate funkiness."
This is a placeholder. I'll include more details of why you shouldn't listen to Threadwatch.org in a bit, and debunk this some. Let me get this posted and I'll follow up.
(Yes, I am GoogleGuy.)
I am really extremely entirely confused about the article altogether. Is the hijacking more or less about Google digging into your site even when your robot.txt crawler robot is refusing google entrance?
The Google index has been a mess the last 6 months to a year. If something like this forces them to get their act together and stop ranking garbage at the top, good.
This is the last straw! I'm going back to MSN, where I know that my data and privacy are being protected!!
*duck*
Socialism: A feeling of discontent and resentment caused by a desire for the possessions or qualities of another.
Google has the records, and probably the original
site exists with behavior dependent on browser name
being GoogleBot or not. The replacement site will
generally have some way of making money, which can
be tracked via financial transactions.
For every Good Thing, there are at least 100 different ways to abuse it.
FLR
I wasn't sure what a 302 hijack was, so here's the obligatory lowdown for those who didn't rtfa (from article linked page) This exploit allows any webmaster to have his own "virtual pages" rank for terms that pages belonging to another webmaster used to rank for. Successfully employed, this technique will allow the offending webmaster ("the hijacker") to displace the pages of the "target" in the Search Engine Results Pages ("SERPS"), and hence (a) cause search engine traffic to the target website to vanish, and/or (b) further redirect traffic to any other page of choice.
arg
see subject.
"I'm just here to regulate funkiness."
A few months ago, I rearranged my website. To make sure people could still find things, I put 301 redirects on all the old pages that I moved.
I noticed in my logs that search engines have repeatedly requested the 301 pages, but often don't follow the links to the new pages. And when searched with google, the pages still show up with the old urls. Should I be using 302 redirects instead?
"Oh! Look! Something beautiful! Something impressive! I must destroy it!"
pah. feeling jaded today, i guess.
"hey, could you pass me a paper towel? er.. I mean... DEPLOY ABSORBTION PANEL!"
buy GOOG on the dip as many non-techie investors panic sell. 8)
Diplomacy is the art of saying, "Nice doggie!" until you can find a rock.
As web presence -defined as within about the first 10-20 results of a search- becomes more and more important to "success," black hat techniques such as this, to eliminate competitors, will become more and more common. Google, or any other search tool needs to be able to stay above the fray and not be subject to hacks such as this.
This is why Gopher will always be better than your feable world wide web junk.
Damn Google!!! Do you mean this is not www.kuro5hin.org ??
I can imagine it now... The slashdotting to end all slashdots. If every site in google was 302 redirected to RIAA.com How amazing would that be...
I told you, I have too much time on my hands!
People are already using the 403 redirect on services such as no-ip and dyndns so they can manage to gain multiple (even whole pages) of listings for the terms they want
Business Voyeur
1. search Google for 'allinurl:', e.g. 'allinurl:slashdot.org'.
/me notices that my company's web site has been thusly hijacked... and yes! Doing a Google search on the main text on my company's web site shows dozens of unrelated sites high in the ranking. None of these actually have the text on their pages.
2. copy and paste any dubious URLS into this tool and check whether they're using 302 redirects or not.
3. Panic!
One example: http://www.tradedoubler.it.
Luckily, the phrase in question is complete gibberish and no-one ever finds our site through Google, only by reputation and word of mouth.
Still, I think it's clear Google have a serious problem here...
Sig for sale or rent. One previous user. Inquire within.
You are right, MSN only sets cookies that last the lifetime of their current OS.
Grundgesetz * 23. Mai 1949 - 30. November 2007 - http://www.vorratsdatenspeicherung.de/
It just is.
/.
It keeps getting harder and harder to find what I'm looking for. I'd say about 3/4s of the links it returns just redirect me to eBay or some assholes Amazon referrer link. If I wanted to deal with that type of sleazy spamming bullshit I'd just read a book review on
Are they doing anything to stop the googlebombing, scamming, and bullshit?
Because I see this bringing Google down eventually, if they dont. I mean, if their search doesnt work, what do they got? A cheesy freeware photo sorting app? Webmail?
I don't need no instructions to know how to rock!!!!
302 hijacks work because Google goes to http://bad.site/ and gets redirected to http://good.site/. It then treats the contents of the bad.site as identical to that of good.site. The effect seems similar to if somebody simply copied an entire page off of your site (I'm not sure if it's actually more serious than this), but it's easier to do because you're just keeping a small table of redirections.
How serious is it? Don't know. It's pretty easy for a webmaster to check for hijacking and have her pages de-hijacked (see aforementioned article). It's probably not as screamingly awful as the threadwatch.org article suggests, but the redirector sites are rather annoying. Several of the comments in the webmaster article suggest that Google has already started moving on the problem.
For at least the last 18-24 months it's been increasingly difficult to find non-spam/redirect/affiliate program links for a search on any popular consumer product on Google. Maybe they have too much faith in their current PageRank and think it needs to be tweaked instead of overhauled. Maybe they think they have enough momentum and don't care. They certainly should have the talent and resources to do something about this and it's kind of sad that they haven't. I predict we'll see another whizzy side project in a few months instead.
The thing is that all they have to do is keep it just good enough that people won't leave. Remember, AdWords is Google's product, everything else [gmail, orkut, etc] they've got is just a way to show you those ads. Google's success is entirely because they had clearly better search results than anyone else. If another company can clearly best them then Google may be in trouble.
301 is a permanent redirect, 302 temporary.
This is why the "302 hack" works. If the redirect is only supposed to be temporary, the search engine keeps the URL of the 302 as the URL for the document, but indexes the content of the page to which the redirect is directed.
301 is what you should be using to point the SEs to your new pages if you've moved them. The behavior is supposed to be for the SEs to replace the old URL in their index with the new one, and furthermore count all links to the 301ed URL as being towards the new one. I don't know why it's not working for the grandparent poster, but it's the way that the functionality is "advertised" for Google and Yahoo, and it should work.
500GB of disk, 5TB of transfer, $5.95/mo
I was thinking that some major crisis had broken out and a million pages were hijacked at once creating something bigger than any other Internet event other, and it caused Google's stock to tank and force to them go private again, lay off workers and go bankrupt. But that's crazy. But still, word it right. Damn it.
In America, you spam computers In Soviet Russia, computers spam you!
It's even worse when you don't even put any spaces between the links. So stop it.
The TFA forgot
16. Profit!
My site the humor archives has been affected by this. I can tell because if you do the following search you can see a bunch of sites that are/were 302ing to my domain. I'm pretty pissed off and I seriously hope Google act soon to rectify the matter.
----
How about adding "Fiction: Google information for webmasters contains any facts"?
Read the fucking article - you don't have to have any access to the victim site to do this - you only need to have a higher pagerank than them.
Everything in moderation, including moderation itself
Is this really as serious as it sounds?
Ruby Neural Evolution of Augmenting Topologies
Domain-vultures - you know, the pr0n companies out there that harvest recently expired domains and point them to their adult content. There are lots and lots of sites like this, where some admin forgot to re-register the site, etc. And then, the domain is held hostage by the pr0n site until a ransom of a couple thousand is paid to them. Just a thought.
Here is more information on this.- 301302-bug.html
http://www.royans.net/blog/2004/12/googles-secert
what major headlines ? millions of pages !! the world is coming to an end !!!!
a quick whois on threadwatch.org (the submitters site) reveals its hosted by search engine spammers
platinax.co.uk which is registed to a UK "company" called BriteCorp
http://www.britecorp.co.uk/
who offer all the usual SE spamming methods
coincidence ?
a whois on britecorp's platinex site reveals they have removed their address from the whois db, and their websites contact details are a mobile phone number (07963 808470)
further investigation on britecorp reveals they are not a "real" company but trading as "Brian Turner" (pic) and companies house dont seem to have any records of any of these companies, though iam sure further investigation could find out more
so why would a supposedly reputable marketing company have a cell phone as a primary contact point ?
something to hide egh ?
or perhaps local trading standards would like to hear about them and their "services" ?
northern scum by any other name
I was on slashdot reading all this stuff when my browser redirected to porn sites......
Oh wait. I got bored and did a search for porn...
I guess that's different.
To continue having the victim's hits redirected, the redirect needs to stay in place, doesn't it?
What in the world does the hijacker gain by having google point him, only to then load the victim's page?
hawk
It will also break many "click trackers", "portals", "directory sites", "search engine optimizers", and other annoyances, which is probably a plus for Google users. You know, those sites where you click on some phrase in Google and, three redirects later, you're at some irrelevant porno site.
If the site is porn and the correct site was something that might attract kids, get the site on that.
There are also some nice laws involving computer misuse. One could argue that Google had been "hacked".
An imaginative prosecuter will have many more ideas.
Why not just fix the bug and then recreate the rankings index? Googlebot hits my sites all the time, so I know that it covers the rest of the internet quite often as well. With their amount of hardware, it probably wouldn't take long.
A musician without the RIAA, is like a fish without a bicycle.
AllinURL returns results where the results are in the URL. So they *should* be returned.
I'm not convinced by this whole 302 nonsense. I haven't seen a single search where a 302 scraper site is ranking above the site it 302s for the scammed text.
To me it sounds like people's sites drop for whatever reason, then they look for a reason and they grasp at this 302 story.
I do an allinurl on my various sites (8 of them) and 6 have scrapers attached, only 1 has disappeared recently and that seems to have been caused by a change of IP address or maybe the loss of the yahoo directory link or perhaps because I have lots of pages with 20-30% similar content.
But if I only had 1 site I could easily blame a 302 problem.
It seems that when page A redirects to B, Google not only considers that a hit for A, but also assigns B's content to A (I just skimmed through all the posts here so maybe that's not what happens).
In that case, it seems to make more sense to just ignore A altogether since the hit and content rightfully belong to B.
This could be done by treating redirects as empty one-link pages, thus unifying the handlers and defeating this practice.
This story does not need "debunking".
What it needs is a rapid and satisfactory answer or Google will find themselves at the receiving end of more angst than they even know is possible.
A concrete example. My company's web site has been in existence since 1995. So we have pretty good page ranking. Our main page has one phrase, very distinct, unique.
When I search for this phrase (in quotes), Google reports hundreds of matches. These sites (except our own) do not contain the phrase but are sites that sell traffic boosting.
The 302 problem is real.
Incidentally, I just spent 15 minutes at Google.com looking for a way to report the problem. Where is that mention of "canonicalpage"? In the bottom shelf of a filing cabinet, behind a locked door that says "beware of the tiger"?
I'm not surprised you got only 30 reports. What I am surprised at is that you appear to speak for Google yet have such an inane response to what is a real (and for many people, a terrifying) problem.
Sig for sale or rent. One previous user. Inquire within.
This has very real potential to be taken advantage of for phishing scams.
Imagine someone searching for their bank's website on Google (because some think that [searching] is how the web works!) and clicking the wrong link. That link takes them to a site that looks just like their bank's website, and maybe there is a security alert on the front page asking them to verify their information. After doing so, they could be redirected to their real bank's site, never having realized their error.
Experience has shown me that most non-techies know they type an address into their browser, but after that, they pay no attention to it which makes this a real possibility.
This is hilarious! Someone please mod up! Hope I get the above mods in M2.
See my journal for slashdot ID's by year. Mine created in 2005. http://slashdot.org/journal/289875/slashdot-ids-by-year
It's about pushing unrelated sites up in the rankings.
For instance: I have a site with excellent page ranking. Now a new site will set up, and do a 302 to my site. Google now gives this new site my page ranking. When the new site is indexed, it removes the 302 redirection.
When you search for my site, you now find these new sites instead. There is no redirection when you click on a link, the the "cached text" that Google shows is wrong.
Basically this technique allows people to get high page rankings without earning them. It's very widespread - I counted over 60 such parasites for my company's web site (which has excellent page ranking).
Sig for sale or rent. One previous user. Inquire within.
And I know two other people who sent one. Maybe you should check again? I doubt me and my mates account for 10% of your responses. If you believe that the people affected by this are all "spammers" then perhaps the problem is false positives for your spam detection filters. In fact you should probably take a look at your spam detection filters anyway. Last time I checked--probably much more recently than you checked for canonicalpage emails, there was a bunch of scraper sites running AdSense where good relevant results used to be.
Email to webmaster@google.com with the keyword "canonicalpage".
Google are not taking this problem seriously.
I'd suggest that if your website is affected, you send an email as above.
Sig for sale or rent. One previous user. Inquire within.
Too bad the parent was rated "flamebait", he/she (She? Yeah right!) has a good point. Results from Google searches have gone to the dumps because of spam, much of it sanctioned by Google. The only real difference here is that these people are not paying Google to spam you. The Google index sucks right now.
This was originally posted the first time a story about this ran, but since a lot of people are still confused, here it is again...
There seems to be a lot of confusion as to why exactly this is such a big deal. A lot of people saying there's no problem or that this is nothing new... basically just not understanding the issue. Let me explain:
Suppose you have a small business under the domain http://xyz.com/, and search engines bring you a lot of traffic because you rank high for keywords in your market. You have a lot of people out there linking to you, a lot of satisfied customers, good content on your site. You're always in the top 10 somewhere when people search for "xyz widgets".
Well, this issue with Google makes it very easy -- incredibly easy -- for someone to knock your site out of the rankings entirely. And I mean for *everything*, to where searching for your own company name in quotes literally buries you hundreds of pages deep in the results. We're talking sites going from getting 1000 unique hits to 10 overnight.
And here's the kicker: It requires absolutely no technical knowledge, no time investment, and is perfectly legal...
All I have to do is have another domain handy that is roughly as popular as yours. And I make a "links" page, like one of those directory services, that lists your website. But instead of being a normal hyperlink, it's a CGI (or PHP or ASP or whatever) script that generates a 302 redirect to your domain... Now, these are very simple, common scripts. One-liners that you can download from cgiscripts.com and stick on your server. The original intent of these scripts is to track which links are being clicked on your site. But now they've found a new use, because when Google gets that 302, all hell breaks loose.
See, according to the HTTP spec, 302 is a *temporary* redirect, which means Google is supposed to interpret whatever content it finds at the 302 target (your site) as really belonging to the URL of the source (my site). Google is just obeying the spec strictly here, and with devestating results. Why? BECAUSE THE DUPE FILTER NOW KICKS IN! You see, Google has a "dupe filter" that says if the same exact content is found for two unique URLs, then one of the URLs is obliterated in the rankings. Because after all, searchers don't want to be finding the same content over and over. If that happens, they'll start using a different search engine. But Google, sticking strictly to the HTTP spec, doesn't know who the content really belongs to when it gets a 302.
So Google essentially flips a coin. And if it comes up tails, say bye-bye to your domain in the rankings. Your *entire* domain. Because the dupe filter isn't limited to just the page that the 302 is pointing to -- it applies across your entire domain.
These 302 "exit-link-trackers" are all over the web. They've been used by webmasters for years. But it's just recently that Google has started treating 302 this way, so it didn't have any bad effect before. But now it kills you.
The funny thing is, the solution seems pretty simple: Just stop treating 302s this way if they point to a different domain. But for whatever reason Google isn't listening. Hopefully the press that's being generated now will give them the kick in the ass that they need.
Okay, so basically this is the problem: when Google encounters a status 302 redirection (as opposed to the status 301 redirection) it then indexes the content as belonging to the initial URL, not the URL at the end result of the 302 redirection. Other things happen later because of google's design.
302 redirections are temporary redirections - the idea is that a 302 is supposed to be used when someone needs to be redirected to a new page, but should still use the original URL if they want to come back later. As an example, the page http://purl.oclc.org/OCLC/PURL/CONTRIBUTORS performs a 302 redirect to http://purl.oclc.org/docs/contributors.html. This means that although your web browser needs to go to some other URL for the content at the moment, they really should remember the first url as the permanent one.
Contrast this with what happens when your browser visits http://snowplow.org/martin - you get sent a 301 redirect to http://snowplow.org/martin/. (Note the extra slash) In this case, the server is saying "the url with the slash on the end is the real location, and you should not try to come back here without the final slash in the future."
Ideally, if every web browser behaved according to spec., bookmarks (remember bookmarks?) would get automatically updated to the new URL when you selected them and the redirect was a 301 redirect. However, for a 302 redirect, the bookmark would stay as is.
302 redirects can be very useful when you want to set up a hierarchy of "logical" URLs that will permanently point to the correct location. 301 redirects are useful when you're obsoleting an old URL and wish people to go and use the new URL from now on.
Okay, so how does this relate to google? Well, let's suppose that you have a great site on fruitbats. I can set up http://www.example.com/topics/fruitbats to be a 302-style redirect to your site, essentially saying "The information at http://www.example.com/topics/fruitbats is temporarily being hosted by http://www.yoursite.com/". Now, google when it spiders pages will see that, will go retrieve the text from your page and will then index it under http://www.example.com/topics/fruitbat, since after all I just gave a temporary (302) redirect.
But it gets worse, because a final part of google's indexing process is to compare pages for identical text, and throw out all but one of the URLs. Apparently this stage has nothing to go on other than the text and the recorded URLs, and so your URL stands a fifty-fifty chance of being thrown out.
Except that I've not just redirected http://www.example.com/topics/fruitbats to your site, but also http://www.example.com/topics/fruitbat, http://www.example.com/topics/fruit_bat, and http://www.example.com/topics/fruit_bats. Now your lone URL doesn't stand much of a chance of being the one kept by the "throw out duplicates" processor, does it?
In a sense, of course, there's little google can do to prevent this, because even if they weighted 302-redirects lower in their "throw out duplicates" stage, I could always just go snag a copy of your website each time googlebot visits, in essence doing the redirection myself. (How? Just search the apache mod_rewrite guide for "Dynamic Mirror") However, doing it through 302 redircts means that google pays for the bandwidth to go get your page, not me. (Not that this is necessarily a signficant amount of bandwidth, since we're only talking about basic google here and not images. Depending on the revenue you get by misdirecting google queries it might be economical)
Of course, for this to really work, I'd need a list of websites sorted by category to build up my redirect db. But wait! The ODP feed provides exactly that.
I am a little bit wary of doi
I'm so thinking of the recent iTMS DRM hack...
This has been possible in php forever. And a lot more hidden than a 302 redirect. You go to one page and depending on where you came from, it shows different content, but the url stays exactly the same. Here, go fool google:
//Content people see that come from google
$referrer = $_SERVER['HTTP_REFERER'];
$findme = 'google';
$from_google = strpos($referrer, $findme);
if ($from_google === FALSE){
echo "Original Content";
}
else{
$content = file_get_contents("http://www.yahoo.com");
echo $content;
die();
}
?>
Googlebot wont see yahoo.com content because it dosent have referrer of google. Or you could do the same thing to googlebot. Get the ip of googlebot and show it different information than whats there.
Hey when are you guys going to make it possible to by music, movies, and tv shows from google? I hearcd rumors to this effect, that you'll have "buy this song" feature. Are they true? You know, you search for a song or tv show and then make it available via a third party or google or something.
That would be cool. I wouldnt mind an interface to sell my songs via google.
He stated that he is an engineer employed by Google. He surfs Slashdot and to some extent speaks for Google, although his membership is paid for out of his own pocket. Someone else, not affiliated with Google, had the user ID before but agreed to transfer it to this guy.
At least, that's what he has said before.
I guess what I haven't seen asked yet is:
Why is this not fixed yet?
C'mon Google.
-- Note: If you don't agree with me, don't bother replying. I won't read it.
Your story would be a lot more convincing if you listed a real site that had been "highjacked" by this technique. Or are you just a spammer?
So you're a Google employee who debunks "myths" about Google but when you say something you don't officially represent Google's opinion and stance on issues, right? How convinient.
I'm surprised nobody has mentioned that Yahoo has already closed the 302 hole.
sigs are a waste of space
Look, there *was* circumstancial evidence for the "Greg Duffy" thing ... i.e. just enough to make it a discussion. I agree that fearmongering is not the way to go. I appreciate that you looked into the issue (and my first instinct is to trust your explanation, that is was a DNS issue).
However, if this is Google's PR method, I think you are kind of asking for it! In the absence of information, the internet community will speculate until the cows come home. I'm not saying it's right, I'm just saying that's reality. Even though I said on my site that I thought Google didn't do anything underhanded I bet a lot of people were still not convinced. Google can do a little better than this, and although you have been fairly nice to me (thanks) this response is a little flamebaity for PR. Please understand that I mean no offense, it's just constructive criticism. Even if everything you say is true, a representative of the company should always at least attempt to sugar coat something like your last paragraph.
Also, on a more personal note, maybe Google should embrace the people that are involved in researching these problems instead of using this broken communications policy. I know that in my case I contacted you guys 5 *months* ago about the Google Print problem I described and never got any followup except for my t-shirt (which I really like). I have some great ideas about possible solutions to the problem I described, and as far as I can see Google has not fixed the root of the problem. When are you guys going to contact me?
-Greg Duffy
My company's web site is imatix.com
You will notice that the site's main page contains very little text. There is one marketroid phrase, "Strategic solutions for a complex world".
Now search Google for this phrase.
Look at the results. A completely irrelevant site has come in at first place. imatix.com is now at second place (this changed today).
imatix.com is an old site, with very high page rank. Now, it does not matter much for us, since no-one is going to search for this phrase, but if this can hit imatix.com, it can hit other sites.
The problem is entirely real, and it is extremely serious. I'd say, if Google don't fix this before it hits the main media, they will suffer irreparable damage to their reputation.
Sig for sale or rent. One previous user. Inquire within.
There is a simple solution for Google: Only honor 302 redirects when the original and target domains match (or points to a subdomain of the original domain.)
In all other cases treat a 302 (temporary) as a 301 (permanent) redirect, thus giving credit for the content to the actual hoster of the content.
This allows webmasters to continue using 302s to setup logical URLs to mask the organization of underlying content but eliminates the ability to hijack completely.
Natural != (nontoxic || beneficial)
Is there a specific search that someone can suggest that would demonstrate this problem?
Here's what I do: Bitty Browser & Andromeda
everyone knows google is #1
being at the top makes you a target and every little gnat is going to chew at you trying to get a piece.
remember altavista and others..
they ended up so spammed you had to go through pages of results to find anything any good.
I just think it has taken a while, but they are catching up with google now.
anime+manga together at last.. in real time.
Taco posts the same URL.
Lars T.
To the guy who modded me down from perfect to terrible Karma - Apple haters still suck
I just did a search for allinurl:mysite and found about.com was stealing some content. Google was indeed pointing readers to about.com.
We're talking more than 30 pages here.
Try http://s.teoma.com/search?q=See%2C+according+to+th e+HTTP+spec%2C+302+is+a+*temporary*+redirect%2C+wh ich+means+Google+is+supposed+to+interpret+whatever +content+it+finds+at+the+302+target+%28your+site%2 9+as+really+belonging+to+the+URL+of+the+source+%28 my+site%29.+Google+is+just+obeying+the+spec+strict ly+here%2C+and+with+devestating+results.+Why%3F+BE CAUSE+THE+DUPE+FILTER+NOW+KICKS+IN%21+You+see%2C+G oogle+has+a+%22dupe+filter%22+that+says+if+the+sam e+exact+content+is+found+for+two+unique+URLs%2C+th en+one+of+the+URLs+is+obliterated+in+the+rankings. +Because+after+all%2C+searchers+don%27t+want+to+be +finding+the+same+content+over+and+over.+If+that+h appens%2C+they%27ll+start+using+a+different+search +engine.+But+Google%2C+sticking+strictly+to+the+HT TP+spec%2C+doesn%27t+know+who+the+content+really+b elongs+to+when+it+gets+a+302.%0D%0A&qcat=1&qsrc=0& Search.x=0&Search.y=0
your sir are an obvious troll, the first link points to slashdot.org/imatrix.com which of course returns a slashdot 404 error page, and the google.com search link returns the imatrix.com websites link rated number one, and a bunch of placeholder sites below it so how does this demonstrate any harm in imatrix.com's page ranking?
come on Mods, at least read the post, and Check the links before you mod up something as informative
Apocalypse Cancelled, Sorry, No Ticket Refunds
When has jurisdiction ever stopped the USA?
We just grabbed a guy out of Australia who'd
never set foot on US soil, unless you count
Australia as US soil.
"Almost Nothing" means "Not Nothing", aka "Ok, yeah, a couple of things". In this case, there's at least one technique that works, and there may be others that nobody's discovered or ranted about yet. But this one's ugly.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
here's my write-up on the problem from early February called Google and the Mysterious Case of the 1969 Pagejackers. the problem has been around for a long, long time.
;)
personally, i'm ready to give up google maps or something else (autolink?) if they would 'fix' this or at least be more transparent about what's going on.
btw, the word on the net is that the googleguy posting here isn't the real one. anybody have details on this?
-kpaul
J-Log: Journalism News, Media Views
I was about to say that the solution was to not replace pages in the index when the redirection happens across domains. But then public hosting sites like angelfire.com came to my mind. So they just have stop replacing redirects altogether.
I'm in a hurry, so can't write much, but:
Couldn't Google do something about this by sending out Googlebots that aren't identified as such, or by requesting the site in another way, and comparing the page served to Googlebot with the page served to the alternate client?
I call it Search Spam. Just as it seems email spam is getting under control stuff like this is getting out of control. I wish there was actual legal punishment for people who do this. Right now it's like a veiled threat. My web searches have layer upon layer of bad sites that have nothing to do with what I concisely searched for. Someone do something!
I'm sure you'll tell me I'm borked to no end, but ...
Let me remind you that it is in no way google's responsibility to drive your web traffic. Google is not your marketing department. That being said - google can do whatever the hell it wants.
It is completely your own problem if the only way you get contact is through people clicking a google link.
If a 302 is intended as a temporary redirect, shouldn't the Googlebot not replace the original index, considering a 302 redirect is supposed to be temporary? Why would anyone ever want what is supposed to be a temporary to be permanently indexed? Google should never have been indexing "temporary" redirects in the first place.
Why all the yammering and discussion on this?
It's pretty simple; 302 redirects allow bad guys to exploit Google.
It doesn't matter that it's the wrong way to use a 302 redirect. They are the BAD GUYS. Remember the "spammers lie" truism?
It's the Google rule that is broken. 302 should be treated as "cant find site" in their search rankings rather than assuming the the data sent by the web server is honest. It sucks that some legit users of 302 won't get ranked as well because of it, but boo hoo. Let anybody that has hardware or software problems get better equipment in the first place if their freaking world ends when they don't get ranked in their keyword group. I have NO SYMPATHY for someone that shoestrings their vital revenue stream infrastructure and then wonders why things go bad. It reminds me of my job too much.
Buy Google ADs if you need to make money off your site traffic.
Google will change the rule or they won't. If they want to stay relevant, they'd better. I find myself getting irritated with Google's crappy search results a lot now days, sooner or later I will find one of the little startup to use and they can kiss off if it keeps up. So I figure they will get to it. They are Google, they are good at what they do.
Now what I think they should do is download snippets of pages via the Google toolbar which then sends the data to Google to make a massively distributed bot-net spider that is indistinquishable from the web-using masses. At that point, as far as exploiting Google via IP of the bot or user agent of the bot IT IS ALL OVER.
Move along, nothing to see here but a bunch of people that don't understand redirect and HTTP protocols.
I have a solution in mind that may or may not work depending on robot/indexer behavior.
What would happen if I installed a filter on my Web server (in pseudocode):
if (!(http.referrer matches(my.domain))
{
send(301, target=full_url)
}
In English, if the client was referred by an external link, immediately issue a 301 redirect to the full URL of the current page. This should inform the robot that it is now looking at a new site and should start indexing content under a URL that I'm positive that I control.
This solution will collapse if the robot refuses to follow the 301 or does something that I don't expect.
I'd be interested to see a response from someone who understands spyders (since I don't).
Not exactly. If you want to learn how to get rid of a 302 hijack, read Google's patents on removing duplicate content:
In a nutshell, Google says that when duplicate content is detected, the site with the highest PageRank wins. Also note that Google now implements LocalRank in addition to their PageRank equation:
The reason a spammer can overtake a site with a 302 is because the spammer is better at building PageRank, LocalRank and keyword relevancy via links to that virtual URL than the legitimate site owner (even if the site has been around for years). A search engine bot cannot determine the age of a link (yet) neither can it determine the legitimacy of a page beyond link popularity and the "revelancy" and "authority" of those links.
The main problem here is that many sites use 302 redirects as a tracking mechanism (which Google tries to parse and follow as links). This makes it undesireable to "penalize" all 302 redirects. These are often good links. Even Froogle uses 302 redirects with its review content, which allowed it to hijack competitor content in MSN Search until a spam report was filed. (Thanks MSN for fixing it so quickly!)
Some solutions needed here:
"Redirects to a page should be treated as having far less PageRank value than the page itself. That will fix the problem."
Won't work. In a nutshell (expounded upon below) -- Google doesn't know what the "real page" is.
Specifically, I generate "301" on my own sites:
http://blurt.org/page_of_files
will generate a 301 redirect to:
http://blurt.org/page_of_files/
which then gets an index of files. Ok?
Now, sometimes I relocate data (popular big files):
http://blurt.org/p_o_f/bigfile.html
would get a 302 redirect to
http://big_ass_isp.com/myplace/bigfile.html
I don't want to loose my "pagerank" over the content location! And I am doing it to myself here...
Google will look at my site, and see the redirect. It spiders the page, and it is under MY url (where it should be).
But, if Google spiders the OTHER site (say, though an automatically generated links page), it may eliminate the proper URL.
But, REDUCING "pagerank" because I am providing for more bandwidth? (which is what you are suggesting). Why on earth would anyone want that?
Now, what SHOULD happen is that the page should have a tag on it that says "Please index me if you accessed me directly" or "Please index me if you accessed me via REDIRECT from there".
That would do it (I think -- not much of a web master, I'm afraid).
As it is, since the big_ass_isp site is under my control, I place a robots.txt file there to prevent Google from spidering big_ass_isp view of my pages.
But, I can't control other people from generating a REDIRECT to me (but I *can* additionally tag the page).
I don't think that there is much that Google can do about this. (Well, they could honour a custom tag - and propose of the W3 folk, which is what I think will happen).
Ratboy
Just another "Cubible(sic) Joe" 2 17 3061
Being the author of TFA that appeared a few days ago, i'll apologize for any confusion - yet, i'd say that you nailed it. Google has one page (as in "a set of indexed content") and a minimum of two URLs associated with it - at least one of these return something else than "OK", or "Not Modified", or whatever. Still, Google manages to pick one of the URLs that doesn't return one of these codes as being the appropriate URI for the set of content.
The interesting thing is that once the average searcher sees a result for, say "Site A" and clicks on it in good faith, he is not taken to "Site A" but directly to a script that is already in place on "Site B" and 100% controlled by "Site B".
Last time Googlebot saw this script, it redirected instantly to "Site A" ("302 Found"), but, you know... Scripts are scripts - they do one thing until you make them do another thing. And if you're a bit smart you can even make them conditional, showing Googlebot one thing and everybody else another. This is not even rocket science, it's really trivial programming at best. All you need is an "appropriate" site to forward "Site A" users to - preferably one that makes you instant money.
I'm not so sure about this though (that's why i snipped it from your post):
Before i wrote TFA that appeared a couple of days ago, i had been writing about this problem on search engine related fora for a very long time - literally more than a year, perhaps even two. These fora are frequented by verified search engine representatives, and the problem has also been solved ...By Yahoo! Not by Google.
It's more serious than simply copying your content, because the new site *replaces* yours in the ranking rather than competing with it. When I set up http://bad.site/1 through http://bad.site/100, all claiming to own the content at http://good.site/, Google displays only one of the 101 options in the listing -- and yours isn't too likely to get picked.
Actually, maybe that would happen anyway if I simply copied your site content perfectly 100 times. Not sure about that. Still, that aspect makes it much more of a concern.
Let's see if I'm understanding this right. Correct me if I'm wrong...
I set up a goat.cx mirror. The goat.cx mirror contains a 302 redirect to slashdot.org, making Google think that my content has been temporarily moved to slashdot.org. Therefore Google thinks that what's at slashdot.org is just a temporary version of what's normally at my website. Therefore people who would have been sent to Slashdot get sent to my site instead, i.e. people trying to find Slashdot via Google get goatse'd.
Correct?
I haven't tried this. It's just an idea knocking around in my head.
What would happen if I set up a stateful filter on my web server that did the following?
1. If the http client provided a referrer header and that header contains my own domain name, exit (and let the request be processed normally)
3. Record the user agent header, client IP address, and current timestamp in some sort of temporary lookup table
4. Issue a http 301 with an absolute URL that points to the current page but with some technically insignificant rewrite from the way that the client requested it. For example, if the request is a simple GET, append a "?" or "&"
If the client was not referred by an internal link, this filter would instruct the client to reload the page in a way that insures that it knows the correct, full URL.
By itself, this would simply cause an infinite loop which a robot would probably detect. That's where the temporary lookup table and slightly modified URL come in. I left step two out of the list above because it does not apply until the second time the agent hits our page:
2. Consult the lookup table. If this agent already hit this page within the last n seconds, exit and allow the request to be processed normally.
I don't know much about how robots such as googlebot behave. I'd love to see a reply from someone who knows more than I do.
Would it be possible for Google to simply disregard all 302 redirects that refer to a domain different than the document being crawled? In this way, all sites using 302 redirects legitimately (referring to their own content in another location on their domain) would be unaffected while the site hijacking scum would be eliminated.
I find laziness to be an excellent motivator.
Sorry for not writing this in the article - it's pretty long already and you just have to cut somewhere, but here goes:
Yahoo was exactly as vulnerable as the rest of the search engines. In fact this problem was pretty bad with Yahoo at one point. What Yahoo did was simply to fix it by implementing some internal rules about how to interpret redirects.
I believe it was fixed around June 2004 - at that time the problem had already been known (and aboused) for a long time, but use was not widespread yet. The details of the fix can be seen on this one-page PDF
It's simple (and identical to the solution i suggest in my article): When "Yahoobot" (actually it's called "Slurp") sees a 302 redirect, it checks if the domains of the redirect and the target are the same. If the redirect is from one domain to another, Yahoo keeps the URI from the target domain. If the redirect is from one page to another on the same domain, Yahoo keeps the "source" (ie. the redirect script URI).
of course, some just have to take your word for it, but i've heard it from other sources too.
;)
;)
in fact, i think i've learned *a lot* today.
re: the k5 article - cool. thanks. i'll try to see if an editor will let me change it later tonight. (and i admit that piece was a bit 'out there' too, although i tried my hardest to be objective for the most part...)
when you go from 7k to 700 visits a day, you start to lash out at things. i think i may have found the real cause of the problem, though. someone was DOSing my site and taking it down. i think Googlebot was trying to hit me before the server could come back up and maybe i got put on an 'unreliable server' list?
in any case, i'm looking into PPC and other options to spread things out some.
thanks again. please don't hold my foolishness against me.
-kpaul (the real one)
J-Log: Journalism News, Media Views
Whilst my original comment was supposed to be slightly tounge in cheek, I shall neverless play you pedantic definition game.
Prior to the get out clause afforded to them by the use of the word "almost" they explicitly state that it is fiction to say a competitor can have another site removed from Google's index.
Either it is fiction, or it can be done. If it is meerly hard to do it comes under the heading of fact.
Absolutely Roflmao!!
:)
:)
:)
I guess some people have never heard of the term "sole trader".
My internet business is barely a year old - almost everything is communicated with other webmasters via e-mail - phone support is provided as a last option, but it means that if anyone really needs to use it, then they can have my immediate attention wherever I am, to have their concerns addressed immediately.
As for spamming - well, this is one of those "anonymous cowards" some of us are familiar with, who believes that if you purchase a link from another site, or become involved in a link exchange, or register your site in a directory - then you're a spammer.
Thanks for the heads up on the Platinax registration details, though - hadn't realised they'd been left out. I had a run in with some Belgian Nazis last year, after I booted them from a forum I admin, when they tried to use it for promoting Neo-nazi propaganda. They've tried a few times to get back at me since, so I've been trying to reclaim some privacy online. Platinax reg details should be public, though - I'll put something online, then try and fine a PO Box for the hate crap.
all emailed google about this problem, like, right now? You think possibly that the wrath of a million slashdotters would make them listen?
Already sent them an emaily.
Try not to let life get in the way of living.
....why would you want to be an investment manager for others in the first place?
it takes more than a little slashdotting... try again: Pagejack article
Report link spam here. I can confirm that I reported a domain full of link spam as GoogleGuy suggested in the March 18th story and the whole mess was quickly removed from their index.
I admit haven't been paying attention to this. What exactly is this 302 exploit? Is it just a matter of attackers spoofing referrer entries in their GET requests so the attacker's web site gets listed on the target's blogroll? Or is there more to it than that?
I'm proud of my Northern Tibetian Heritage
An example was posted in the beginning of the thread: site:drudgereport.com
A quick count showed 12% of the top 100 not being the real domain (i may have missed one or two). Actually this is quite common for the major news sites (please disregard opinion on drudgereport, this is about his URLs not his journalism)
And, clsc.net is still not down :p
As also written in TFA: The search engine spiders don't send a referrer, so your method won't work. No, they can't "just send a referrer", because they could have found a link to your site on a lot of pages, so which one should they choose? Also, some popular firewalls don't send the referrer either.
(see title of post)
Muppet.
LOL All these posts to explain the same thing over and over. The sad part is..there are others that still don't get it.
Freenetworks.org used to do 302 redirects to "affiliate" sites.
ie. http://amsterdam.nl.freenetworks.org goes to http://www.losnet.nl
Freenetworks.org has a pretty high pagerank in the wireless community, so they ended up hijacking sites where the url of the affiliate site would be replaced by xxx.xx.freenetworks.org instead of the usual url. This was fixed when freenetworks were notified and they changed the 302 redirects to 301's.
i didn't see it in the first few hundred of a site:domain.com
what's the other explanation for another URL having my title, description and cache?
when i run a header check on it, it shows a 302 redirect then 200.
in this case, i think the other person doesn't know using 302 isn't correct (ie. it's a link collection script...)
is there another explanation?
-kpaul
J-Log: Journalism News, Media Views
A woman of age 50 can be a grandma easily, (ie child at 23, gives birth at 20). That grandma grew up in the 60s/70s, and most likely went to clubs and hanged out with the hippies etc... so 21st grandmas are all hip and cool not like the yesteryear pre 40s teeners.
Liberty freedom are no1, not dicks in suits.
"is supposed to interpret whatever content it finds at the 302 target (your site) as really belonging to the URL of the source (my site)."
Claiming ownership of someone elses copyrighted works, I would think is actionable.
good luck in seeking a legal remedy though.
'allinurl:' shows URLs that contain a specific keyword, which can lead to false positives. 'site:' is supposed to show only the pages that Google knows about within a certain domain. If you search for 'site:yoursite.com' and get results from sites other than yoursite.com, then you know you've been affected. Especially if those other domains have taken the #1 result.
Here is one example.
Here is another example.
Thanks to the War on Drugs, it's easier to buy meth than it is to buy cold medicine!
kpaul i'll locate you elsewhere.
/. threads aren't linear - new messages pop up everywhere, including the parts you have already read once).
- i'm not sure if you re-read these threads (it's a godd thing to do as
If you do, and for others:
The "site:example.com" search is a good tool. However, it's not always practical if you have a lot of pages, as it's not always that you will be able to spot hijacks among the first 1000 pages.
So, try searching for specific document titles in stead, putting the document title in quotes. This way you will easily see if there's a result that has your headline, your snippet, your cache, and a URL that is not from your domain.
>> what's the other explanation
In all this talk about 302's we sometimes forget that a META REFRESH with a timeout of zero can do the exact same thing as a 302 redirect.
also, it coud simply be one of these:
- a copy of your page(s) on another domain
- a mirror of your page(s) on another domain
- another domain proxying your domain
Regarding these three cases, the "wrong URLs" should not be seen as an error, imho.
When i use these words, "mirror" usually refers to "close to verbatim copy" (more than 95% verbatim copy) -- ie. almost no difference in the content from your page -- while "copy" could easily just be fragments of your page, perhaps even mixed with fragments of other pages. A "proxy" will be 100% verbatim copies; in fact it will be your exact site, only shown on another URL.
For those that follow these things, 1bu.com is not a proxy, it's a mirror (as it strips out flash and stuff).
The webmaster of doi.contentdirections.com
and doi.org can agree to add some extra
robot rules that tell googlebot
that it's ok to follow 302 redirects
to/from doi.contentdirections.com and doi.org
another solution is for webmasters
of doi.contentdirections.com to only allow
redirects from doi.org, so the 302 trick don't work
get it?
I am seeing all over the net a discussion on 302 Hijackings and that Google is evil. But the thing is no one is discussing the actual cause of it. The actual cause is the HTTP Protocol that says EXPLICITLY
:00:00 GMT
"10.3.3 302 Found
The requested resource resides temporarily under a different URI. Since the redirection might be altered on occasion, the client SHOULD continue to use the Request-URI for future requests. This response is only cacheable if indicated by a Cache-Control or Expires header field." - Emphasis not mine.
You can read it for yourself at http://www.w3.org/Protocols/rfc2616/rfc2616-sec10. html
Now we all know the importance of protocol. Its a communicating language. In this case the protocol was basically developed when the WEB was pure and unadulterated. Where people expected others to follow and not misuse the protocol.
But with money always comes greed and dishonesty. WEB originally was not built with Business in mind. It was for free Information Interchange. But it has just evolved to a state where Commercially the WEB can be harnessed (exploited whatever) for its potential.
So now any search engine that follows the protocol to the letter is in effect aiding the Hijacking, but is it the mistake of the search engine or the protocol? Unlike Human languages, protocols dont evolve uninhibited. If it did then very soon no browser can understand all the servers and vice versa. i.e you might need 10 kinds of browsers to access 10 different website, beacuse those 10 websites talk a different language.
(Now come to think of it, is this not what is happening in the DRM world. You download music from one site and you can't play it on another without a hack). That is the reason there is a standard and it gets revised every so often so that it can also keep up with the times.
So some of the suggestions like throw the redirecting page into the bin and keep the target page will really have web wide repurcussions for people who use it with the standard in mind and with a legitimate purpose. So you ask who uses it and for what purpose?
Let me give an example.
Ever tried buying from Amazon.com?
Okay how do you reach the homepage?
Well i type in amazon.com into my browser and i get the page. BUT the url at which i get the page is exactly now http://www.amazon.com/exec/obidos/subst/home/home. html/103-7996157-2162261
Use this server header tool for understanding what happens http://www.webrankinfo.com/english/tools/server-he ader.php
1) Enter www.amazon.com
It says
HTTP/1.1 301 Moved Permanently
Date: Thu, 24 Mar 2005 14:38:22 GMT
Server: Stronghold/2.4.2 Apache/1.3.6 C2NetEU/2412
(Unix) amarewrite/0.1 mod_fastcgi/2.2.12
Set-Cookie: skin=; domain=.amazon.com; path=/; exp ires=Wed, 01-Aug-01 12:00:00 GMT
Location: http://www.amazon.com:80/exec/obidos/sub st/home/home.html
Connection: close
Content-Type: text/plain
So amazon.com doesnt exist (dont mistake me, the page amazon.com) what exists is http://www.amazon.com:80/exec/obidos/subst/home/ho me.html. 2) Now enter http://www.amazon.com:80/exec/obidos/subst/home/ho me.html in the box.
It says
HTTP/1.1 302
Date: Thu, 24 Mar 2005 14:40:48 GMT
Server: Stronghold/2.4.2 Apache/1.3.6 C2NetEU/2412
(Unix) amarewrite/0.1 mod_fastcgi/2.2.12
Set-Cookie: session-id-time=1112256000; path=/; do main=.amazon.com; expires=Thursday, 31-Mar-2005 08
Set-Cookie: session-id=002-8272699-5270422; path=/ ; domain=.amazon.com; expires=Thursday, 31-Mar-200 5 08:00:00 GMT
Location: http://www.amazon.com/exec/obidos/subst/ home/home.html/002-8272699-5270422
Connection: close
Content-Type: text/html
So now the home page is temporarily at http://www.amazon.com/exec/obidos/subst/home/home. html/002-8272699-5270422
If Google were
Lets say you could beat the index pretty soundly, acheiving a reliable 15% annual return. Let's further say you have considerably more available capital than most, say 1 million dollars. So you can manage your money and make 150,000 in a year, and you probably should.
Why would you want to be an investment manager for others? Because if you can reliable acheive a 15% return, "others" will pay you several million a year, at least.