Slashdot Mirror
Google De-indexes Talk.Origins, Won't Say Why UPDATED
J. J. Ramsey writes "Talk.Origins is an archive with thousands of pages exposing creationist pseudoscience. Rather mysteriously, Google pulled the plug on its search engine, giving only the vague reason: 'No pages from your site are currently included in Google's index due to violations of the webmaster guidelines.' This was apparently triggered by a recent cracking of the site that added 'hidden links to non-topical sites,' but Google won't say just what the violations were. Talk.Origins webmaster Wesley R. Elsberry believes that this Google policy harms honest webmasters." From the article: "My mission, whether I liked it or not, was to find and fix whatever problem the [Talk.Origins Archive] might have, with no guidance as to what the problem was and nothing at all about where to start looking... I was extremely lucky. The damage to my site was limited and in the first place that I happened to look. Other honest webmasters might not be so lucky. They may have to undertake an arduous process of vetting pages, essentially having to second-guess the mind of the cracker in trying to locate a problem that Google knows the exact location of." Thanks to an alert reader who sent in Matt's blog posting about how Google handles hacked sites.
Nobody was evil here. The guy's site got hacked and spam links added, Google rightfully de-listed him, and then the webmaster found the problem, fixed it, and asked Google to re-list. Am I missing something?
The writeup sucks. It implies that Google is censoring Usenet.
Shop as usual. And avoid panic buying.
was that he had no idea why he was delisted so he could fix the problem.
"exposing creationist pseudoscience"...
Slashdot is so biased I don't know why I even bother anymore. Bashing Christians is so fashionable these days.
"Creationist" != "Christian", but don't let that stand in the way of your pretending to feel victimized.
Shop as usual. And avoid panic buying.
Sounds like you blew the cover there, dude.
So many people refer to Google as if it were a human looking at web sites and giving it the big thumbs up or down. As part of the indexing if the spider finds "violations" such as presenting a different page to spiders than to humans, it risks being dropped from the index. To expect a human response to why each site triggered the de-indexing is not reasonable.
In the webmaster's whining about Google, he complains about the request to be re-indexed containing:
*I believe this site has violated Googles quality guidelines in the past.
* This site no longer violates Googles quality guidelines.
He thinks these are "an admission of guilt", but they dont' say "I violated" they say "the site violated". So, if the site were hacked and did violate their indexing policy, fix it, say you've fixed it and move on. How many hits has he had over the years that came directly from Google? And did they come from Google due to all those people choosing Google to search for his site or it's topics? But now he whines about being delisted for the time it takes him to fix a site he should have kept unhacked in the first place.
People may be treating Google as a public utility, but Google (a private company) has absolutely no obligations to any website.
Ultimately, Google* has the right to change the rules when & if they please, in an arbitrary fashion, without consulting anyone.
*When I say "Google" I mean "the guys who own a majority stake in the company and cannot be overruled"
[Fuck Beta]
o0t!
I was under the impression that they told the webmaster the reason they were delisted, they just didn't tell the webmaster the specific pages that the reason pertained to. Like "Your site has been delisted for hidden links to non-topical sites" instead of "Your site has been delisted for hidden links to non-topical sites on pages index.html, intro.html." etc. To me, that's a webmaster job. Google did their job on their end. What if the site had hundreds of pages of non-topical links? What if Google spiders just stopped at the first one they indexed (as they should). Should google be in charge of going through this guy's site and telling him exactly where the problems are? They are a search engine, not a website security firm. People are getting lazier everyday and everyone expects someone else to do their dirty work for them. People need to take some responsibility and stop whining.
Unfortunately you're missing something too.
Google is in an arms race with spammers and blackhat seo firms. How are they supposed to know whether someone is honest or just mining them for information for their scam?
Gamingmuseum.com: Give your 3D accelerator a rest.
Public utilities for a town have certain responsibilities only because they have accepted those responsibilities in exchange for the town making them a monopoly.
Google has no such responsibilities just becuse of the way they're treated by users. (And even if you argue that they're a monopoly, they haven't been granted monopoly status by a government.)
What you're missing is that Google gave him no clue/hint/guide/comment/help on why he was delisted.
I'm not for censoring any information, and I am not trying to defend google. But there may be one very good reason why this may be happenning this way.
Google is at war with search engine spammers. When google de-lists somebody for spamming their search engine, if they gave a specific reason why then all the spammers would do is tweak their spam farm and be up and running in a couple of hours.
If they told this guy what was wrong, they would have to spend a huge amount of time and resources telling why everyone is wrong, all the while helping out the spammers.
Google is a good search engine, but if you notice that if you go beyond a couple of pages out of search results, many times you will find nothing but useless "link farms." Unfortunately, spam is no longer limited to email inboxes anymore, it's everywhere.
Take the cheese to sickbay, the doctor should see it as soon as possible - B'Elanna Torres, "Learning Curve"
Bzzt. The website admin needs to locate one or more problems (== however many the cracker planted), and Google knows the exact location of at least one. "one or more" >= "at least one". If google tells people where their problems are, google will be playing whack a mole for eternity. There are contractors/services that should be able to help them/anyone, google is not one of them.
ustr: Managed string API with ave. 44% overhead over strdup(), for 0-20B
With the index sizes that are being collected by search engines these days (on the order of 10 billion entries), it's completely naive to think that some humans are sitting at a terminal choosing to delist websites for some policy reason or other. It's also completely naive to think that a human email monkey can do any sort of digging to find out the exact reason that Google's automated algorithm has censored this particular site.
Instead, Google's engineers have automated algorithms which do all the censorship, and the policy is just there as a thin cover for whatever the algorithm happens to be doing today. It's worse of course, because 1) algorithms change every few months and 2) there's simply no comprehensive way to test the quality of the implementation.
Anyone who's programmed a nontrivial algorithm knows that obscure edge cases are a bitch, and with 10 billion websites, any algorithm will have plenty of obscure edge cases which nobody has ever tested, nor ever will. The most likely explanation is that the website in TFA is a false positive of some subsystem, but fixing it will require changes to the algorithms, and Google don't want to risk that, would you? The problem will probably go away in a few months when the algorithms are scheduled to be updated.
ID does not propose that the creator must be a diety
ha ha ha ha. Yes, in ID the creator must only be someone eternally existing with the ability to manipulate all matter in the universe at will.
But diety [sic].... no!
In case you missed it, in ID it must be a deity, or else who created the creator? If life can not come from non-life, then there must be some eternally existing intelligence to kick things off (aka God). So either you don't understand the theory, or you are lying.
You have to love when a theory tries to sound more sane by saying "but... it could be space aliens too!"
Is there anything I'm missing there about ID?
It might be good, but my point is that Google doesn't have to... and maybe shouldn't.
To some extent, part of Google's ability to foil bad website behavior relies on security through obscurity. If Google doesn't tell or hint to anyone how the cheat-detecting algorithms work... well, isn't that good for Google?
I could make the argument that since (as you argued) Google is a public company, they have to do what's best for the shareholders by doing what's best for Google. But that is an irrelevant argument, since there's really only three people whose opinions on the subject matter.
If Google ever did do something along the lines of what you're proposing, they'd have to put a lot of time & effort into setting up a system that can't be easily abused by link spammers, is easy to use for idiots, etc etc etc.
That may be more trouble than it is worth, compared to saying "not our problem, deal with it yourself."
[Fuck Beta]
o0t!
> Google is at war with search engine spammers. When google de-lists somebody for spamming their search engine, if they gave a specific reason why then all the spammers would do is tweak their spam farm and be up and running in a couple of hours.
Security through obscurity is no security at all. The spammers already know Google's weaknesses -- that's why there's so much spam everywhere.
My other car is first.
People may be treating Google as a public utility, but Google (a private company) has absolutely no obligations to any website.
PG&E is a public company. ComEd is a public company. Verizon is a public company. AT&T is a public company. They're all public utilities. Simply being a publicly traded for profit corporation doesn't mean that you're not a public utility.
Ultimately, Google* has the right to change the rules when & if they please, in an arbitrary fashion, without consulting anyone.
Yes, but there is something called ethics. Google is held to a higher standard than the Ackbar and Jeff's Falafel and Oil Change Hut because of their unique position of being depended on by hunderds of millions of people the worldwide. Also, Google said they should be held to a higher standard with their "Don't be Evil" slogan.
Did Google act wrong in this case? No. But that doesn't mean that your larger point about corporations are beholden to no one is valid.
Basically, this "so called" webmaster wanted free consulting from Google. I don't think so. My personal response would have been, "I'll be happy to supply you with the information you request. It will, however, cost you my standard consulting rate of $xx/hour, two hour minimum."
Only friends and family get free computer help from me, but I'm rethinking that policy since I spent half a day cleaning the malware off my brother's computer during the last family holiday. He probably won't ask me to do it again, though. When he asked how his system got so infected, I answered (in front of the entire family), "You got infected from all those lesbian porn sites you've been visiting."
-- Will program for bandwidth
If you say that that's a metaphysical question that cannot be answered, why not just skip the whole designer/creator bit and say that you are not interested in physical modeling of the world. Invoking an extremely improbable super-being to explain the world is very unhelpful. That's what earlier civilizations did: thunder was Thor riding in his carriage in the sky etc
What the ID followers want is a return to that using the logic "I don't understand it so it must be God's work."
Google has been up front with where their loyalties lie in the search engine business: With the user. They got big and continue to be big because the give results that the search users are looking for. In general, this means the links they present are on the topic queried for and on the basis of links from other sites the content has been "rated" useful.
If a site is designed ( or screwed up ) such that it shows as a result to a query when inappropriate, delivers spam, or ranks higher than the content would warrant, and Google still presents it as a search result, then Google has failed their customer.
Webmasters are not their customers, individuals who are searching are. Ethics says that you give your customers what you promised them. Ethics says you live up to what your stockholders expect by doing what you told them you do: Delivering search results that keep your customers coming back ( and serving them up ads each time ).
Going on the market puts some control of the company into the hands of shareholders, not the general public. Become a shareholder, then you can have a say and ask for a nice, friendly email.
I love my sig.
It's funny how the Google apologists are always around on Slashdot to defend Google's (a private company) right to screw anyone, ...
It's also funny how the Google haters are also here to throw stones at every little perceived problem with how Google works. It's funny how they also seem to use Google a lot, despite of them. I wonder why that is. Could it be that Google does what they want it to do? And, is it Google's problem that so many sites have come to the conclusion that their very existence is tied to their Google page rank? If you do not like how Google works, don't use them, and use your site's robots.txt file to exclude them from indexing your site. The more people who use other search engines, the less "power" Google (or any other search engine) has over "the market".
In this particular case, Google gave the webmaster sufficient information to discover the problem. If it wasn't enough for "other honest webmasters", then they aren't particularly competent, in my opinion, which would tend to affect how I felt about their information being relevant, too. A lot of people spend a lot of effort trying to scam their way to the top of the page ranks. And it looks like Google is spending a lot of effort to keep the game "honest".
Google has no stake in my using their service, other than wanting to display advertising to me, just like a TV or radio station. Given that the website in question here is not a paid advertiser on Google, I don't see where they have a responsibility to do anything special for them. Their responsibility is to make money for their stockholders, the same as any other corporation. Their "niche" for doing this is to sell advertising that is displayed to people who willingly come to their site. Their way of making people come to their site willingly is to index pages in as "honest" a way as they can figure out to do. Refusing to index a particular site for dishonest links, whether intentional by the owner or not, makes them more desirable to most of their users.
And a few dozen people bitching about it in a front page story on Slashdot doesn't hurt, either.
> There are a lot of hypocrites on that site. They claim that religious people are closed minded while completely ignoring anything the other side presents out of hand.
Can you call our attention to any creationist claims that have ever been made on talk.origins that didn't deserve to be dismissed out of hand?
> This blind faith in popular theories is not just restricted to theoretical physics but also appears in the biological sciences as well. Science is supposed to be a tool for discovery. It is not supposed to supply the meaning of life
Biology is no more concerned with the meaning of life than geology or meteorology is.
It's just that some peoples' world views are threatened by the facts that biology has uncovered.
> or delve into things which are best left to philosophers and theologians given our current state of technology.
I don't know of any question best left to philosophers and theologians. If it's not supported by evidence, it's just someone's opinion.
Sheesh, evil *and* a jerk. -- Jade
Have a peek over at the forums at WebmasterWorld, DigitalPoint, SearchEngineWatch, or any number of other webmaster related sites. This happens all the time. It is an issue that webmasters have had to deal with for some time now. Google at least provides some input for you if you can be bothered to register a sitemap with them.
Google has several billion pages in their index, and a significant portion of them are spam. Their business model relies on them having internal methods of dealing with web spam and it is not feasible or desirable for them to produce a list of violations to each and every person who runs afoul of their algorithms.
This is far from the most popular or important site this has happened to. Wordpress was delisted, as was BMW, Syndic8, and many others. This guy is using the controversial nature of his subject matter in an attempt to draw more attention. Get in line buddy, there is a long list of people whining all over the web about the same thing. Are you more important because the word Christianity is loosely affiliated with your site? Nope.
Do a little googling yourself and you can pretty easily figure out how to resolve the problem. It takes some time, and there are ways to accelerate the process. If you are that reliant on Google, it is time to start participating in some webmaster communities and figure out how to play ball with the Search Engines. Just like everybody else.
But if they tell the webmaster, who might be cheating, (remember, a lot of the exploits out there are actually used by the webmaster) where the problem is, then the cheating webmaster only has to get rid of one exploit and gains insight into the detection methods employed by Google. Then he can leave all the others in place. Wouldn't it be fair to say that the people doing evil is, well, the exploitive webmasters?
Don't hit reply yet...I know this guy was honest, but how in the hell could Google possibly tell who is legit and who is not? Google can't hope to be "fair," only just.
An important change for education.
It would be the same as Microsoft stopping an application from running under windows.
Which would be an entirely appropriate response if said application was a virus.