Slashdot Mirror

In 2004, when Google users discovered that the top search result for the word "Jew" was the anti-semitic site Jew Watch, Google ran a disclaimer in the space usually reserved for ads, explaining that their results only reflected the reality of link counts on the Web, and that they did not endorse any Web sites which appeared at the top of their listings. Now the disclaimer has been dusted off again, as the top result on Google Images for "Michelle Obama" is a picture of a monkey's face with Michelle's hairdo. (Ironically, it looks as if the original image would have fallen out of the rankings, if it hadn't been for a follow-up blog post about the controversy, which itself now comes up as the first result.)

I first heard about the controversy from Dennis Prager's column in which he takes a New York Times columnist to task, because the columnist complained about "racially offensive images of the first couple" that come up in Google searches. Prager was unable to find any examples from Googling "first couple" or "Michelle and Barack Obama pictures," so he concluded that the NYT columnist "wildly exaggerated, if not made up" his claims. I tried Google Image searches for "first couple," "Barack Obama," and some other terms, and I couldn't find anything controversial either. However, it only took 10 seconds to enter "first couple google images controversy" on the regular Google Web search and find multiple blog posts explaining what all the fuss was about. Back to Google 101 for Dennis.

Many of the blog posts refer to Google's disclaimer about not tampering with search results. Those on one side are urging Google to make an exception and "fix" the results, while others sagely observe that Google just reflects reality, it doesn't create it.

All of this punditry is starting from a premise that's wrong. Google has actually removed pages from their search results — not because the pages were illegal or because the webmasters were search engine spamming, but because of the page's "offensive" content. In the "Chester's Guide" incident, a councilman in Chester, England discovered that one of the search results for "chester guide" was a satirical page titled "Chester's guide to picking up little girls." Although the page itself was obviously just someone's idea of sick humor, a Chester city councilman (who admitted that he hadn't looked at the page, saying that the title told him everything he needed to know) urged Google to remove the page from their index. Google at first refused, but later manually blacklisted the page to prevent it from appearing in their search results.

Whether or not you think this was the right decision, probably depends on what you think is the purpose of Google. If Google's purpose is to return the most useful results, then it made sense to remove the link, as Danny Sullivan of Search Engine Watch argued at the time, since it almost certainly was not a useful result for people searching for "Chester Guide." On the other hand, if the primary purpose of Google is to reflect the reality of what pages on the Web feature certain words most prominently (combined with all the other factors that Google weighs, of course), then the results shouldn't be altered.

But more people should at least realize that it happened. The Google disclaimer doesn't precisely say that they never blacklist pages or modify search results ("Google reserves the right to address such requests individually"), but it seems to give most people the impression that that's the case. According to that crudest of Googling techniques for which novice searchers are so frequently lampooned, there appear to be about 400 times as many stories on the Web about the Google "Jew Watch" controversy (where Google stood their ground) as there are stores about the "Chester's Guide" incident (where Google caved).

And Google-number-three Matt Cutts posted on his blog back in March explaining why Google does not remove "offensive" pages from search results; over a hundred comments followed, debating the pros and cons of the position, but none of them mentioned the Chester incident or any other case where Google actually had removed pages except as a result of a court order. One isolated comment from "Anonymous" said:

This is not quite true. I know of at least one web site that was de-listed for containing illegal content and/or promoting illegal activity.

which may or may not have been a reference to the Chester Guide incident. And that was it.

Is this a lot of hay to be making over something that happened years ago? Well, for one thing, I doubt if it happened just once. Consider that the Chester Guide incident involved a public declaration of outrage by a city council, and a public statement from Google, and still hardly anyone knows that it ever happened. If other incidents occurred without those high-profile elements, it would be even harder to discover them now. We'll probably never know how many such incidents took place, unless someone sues Google (maybe the owner of a blacklisted website, or maybe the victim of a RipOffReport hatchet job wondering why that site hadn't been blacklisted long ago), subpoenas Google for a list of cases where pages were de-indexed, and publishes the list if it's not sealed by a court order.

But whether it was one time or a handful, consider that political candidates like Arnold Schwarzenegger and Al Franken got asked during their campaigns about things they did 20 years earlier, and it's fair to ask a candidate about their past, because it's the same person standing in front of you now. Why did you do that? Have you stopped? Why?

And in the big scheme of things, Google is probably more powerful than a single US senator or the governor of California. So, can't we ask? What are their real rules about page removal? Have those rules changed since the Chester's Guide controversy? Can they even tell us what their rules are, or do they consider it a trade secret?

It is well known, of course, that Google censors some results in their search engines branded for different markets like China and even in liberal democracies like Germany. But nobody would call that a slippery slope towards censorship in the US version of Google, because the censorship in the Chinese and German versions is done at the behest of the governments there. On the other hand, Google does admit that they will de-index pages which include credit card numbers or social security numbers (which are all too easy to find on the Web). This might not seem like a controversial position, but even this act of voluntary self-censorship may be dipping their toe in the water further than it seems. Most people do consider their credit card information more private than their home address. But surely there are people like J.D. Salinger who less about the privacy of their credit card number (which is easily changeable) than their home address (which isn't). If someone finds Salinger's address and posts it on the Web, should Salinger be able to demand that Google de-index the page? Why should Google cater to the majority who want to keep their credit card number secret, but not to the minority who care more about keeping their address secret? Another commenter on Matt Cutts's blog post asked:

"hi. I have a question. My mom 'googled' herself and it shows some of her medical problems. She wants/needs these pages removed from search engines."

Again, why shouldn't that be considered at least as private as a credit card number?

And finally, even Google's decision to display an "offensive results" disclaimer, for some results but not for others, raises the same "Where do you draw the line?" questions as the issue of page removal. The Michelle Obama monkey picture gets a disclaimer. But search for 'george w bush' and the first row includes a photoshopped (I think!) image of Bush flipping off the press. Does that warrant a disclaimer as well? (Maybe that's considered less unfair because, even though the picture is fake, it does depict something that actually happened.) The first image result for "bristol palin" is a photo of her engaged in underage drinking — a real photo, but probably unfair to call it the single most relevant photo of her on the Web.

So while Google might consider credit cards and social security numbers and search engine spam to be on one side of a "bright line," and everything else is served up without alteration, I think the line is blurrier than that, for at least those three reasons: (a) credit cards and SSNs are less private than some other that things that Google serves up anyway; (b) Google has unambiguously removed some content that fell outside that bright line, as in the Chester's guide incident, and (c) they make other "slippery slope" judgment calls about search results all the time (as in the question of when to show the disclaimer). So I hope that Google someday comes out with a more complete answer to the question. What is their real policy on what they will remove? The Chester's guide incident — would they do that sort of thing if the same situation came up today, or have their rules changed? If they want to go really deep, then is there a general set of principles from which their rules follow — explaining why, for example, they treat credit card numbers as more private than sensitive medical information? (Google did not respond to my request for comment, either through official channels or the unofficial back channels of friends who work there.)

I hope Google gives an answer some day. Even just to say, "It's a classified internal policy and that's all we're going to tell you." But once and for all, the answer is not "Google doesn't remove content just because it's 'offensive' or 'harmful.'"

Meanwhile, a modest suggestion about the disclaimer displayed above the search results: Put it where people will actually see it, in a separate line below the ads, but above the search results. Right now the link to the disclaimer is displayed as one of three ads across the top, and people don't look at the ads. But hey, people do buy ads, so if you push the disclaimer down a bit where people will read it, you also free up space for 50% more ad revenue!

In 2004, when Google users discovered that the top search result for the word "Jew" was the anti-semitic site Jew Watch, Google ran a disclaimer in the space usually reserved for ads, explaining that their results only reflected the reality of link counts on the Web, and that they did not endorse any Web sites which appeared at the top of their listings. Now the disclaimer has been dusted off again, as the top result on Google Images for "Michelle Obama" is a picture of a monkey's face with Michelle's hairdo. (Ironically, it looks as if the original image would have fallen out of the rankings, if it hadn't been for a follow-up blog post about the controversy, which itself now comes up as the first result.)

I first heard about the controversy from Dennis Prager's column in which he takes a New York Times columnist to task, because the columnist complained about "racially offensive images of the first couple" that come up in Google searches. Prager was unable to find any examples from Googling "first couple" or "Michelle and Barack Obama pictures," so he concluded that the NYT columnist "wildly exaggerated, if not made up" his claims. I tried Google Image searches for "first couple," "Barack Obama," and some other terms, and I couldn't find anything controversial either. However, it only took 10 seconds to enter "first couple google images controversy" on the regular Google Web search and find multiple blog posts explaining what all the fuss was about. Back to Google 101 for Dennis.

Many of the blog posts refer to Google's disclaimer about not tampering with search results. Those on one side are urging Google to make an exception and "fix" the results, while others sagely observe that Google just reflects reality, it doesn't create it.

All of this punditry is starting from a premise that's wrong. Google has actually removed pages from their search results — not because the pages were illegal or because the webmasters were search engine spamming, but because of the page's "offensive" content. In the "Chester's Guide" incident, a councilman in Chester, England discovered that one of the search results for "chester guide" was a satirical page titled "Chester's guide to picking up little girls." Although the page itself was obviously just someone's idea of sick humor, a Chester city councilman (who admitted that he hadn't looked at the page, saying that the title told him everything he needed to know) urged Google to remove the page from their index. Google at first refused, but later manually blacklisted the page to prevent it from appearing in their search results.

Whether or not you think this was the right decision, probably depends on what you think is the purpose of Google. If Google's purpose is to return the most useful results, then it made sense to remove the link, as Danny Sullivan of Search Engine Watch argued at the time, since it almost certainly was not a useful result for people searching for "Chester Guide." On the other hand, if the primary purpose of Google is to reflect the reality of what pages on the Web feature certain words most prominently (combined with all the other factors that Google weighs, of course), then the results shouldn't be altered.

But more people should at least realize that it happened. The Google disclaimer doesn't precisely say that they never blacklist pages or modify search results ("Google reserves the right to address such requests individually"), but it seems to give most people the impression that that's the case. According to that crudest of Googling techniques for which novice searchers are so frequently lampooned, there appear to be about 400 times as many stories on the Web about the Google "Jew Watch" controversy (where Google stood their ground) as there are stores about the "Chester's Guide" incident (where Google caved).

And Google-number-three Matt Cutts posted on his blog back in March explaining why Google does not remove "offensive" pages from search results; over a hundred comments followed, debating the pros and cons of the position, but none of them mentioned the Chester incident or any other case where Google actually had removed pages except as a result of a court order. One isolated comment from "Anonymous" said:

This is not quite true. I know of at least one web site that was de-listed for containing illegal content and/or promoting illegal activity.

which may or may not have been a reference to the Chester Guide incident. And that was it.

Is this a lot of hay to be making over something that happened years ago? Well, for one thing, I doubt if it happened just once. Consider that the Chester Guide incident involved a public declaration of outrage by a city council, and a public statement from Google, and still hardly anyone knows that it ever happened. If other incidents occurred without those high-profile elements, it would be even harder to discover them now. We'll probably never know how many such incidents took place, unless someone sues Google (maybe the owner of a blacklisted website, or maybe the victim of a RipOffReport hatchet job wondering why that site hadn't been blacklisted long ago), subpoenas Google for a list of cases where pages were de-indexed, and publishes the list if it's not sealed by a court order.

But whether it was one time or a handful, consider that political candidates like Arnold Schwarzenegger and Al Franken got asked during their campaigns about things they did 20 years earlier, and it's fair to ask a candidate about their past, because it's the same person standing in front of you now. Why did you do that? Have you stopped? Why?

And in the big scheme of things, Google is probably more powerful than a single US senator or the governor of California. So, can't we ask? What are their real rules about page removal? Have those rules changed since the Chester's Guide controversy? Can they even tell us what their rules are, or do they consider it a trade secret?

It is well known, of course, that Google censors some results in their search engines branded for different markets like China and even in liberal democracies like Germany. But nobody would call that a slippery slope towards censorship in the US version of Google, because the censorship in the Chinese and German versions is done at the behest of the governments there. On the other hand, Google does admit that they will de-index pages which include credit card numbers or social security numbers (which are all too easy to find on the Web). This might not seem like a controversial position, but even this act of voluntary self-censorship may be dipping their toe in the water further than it seems. Most people do consider their credit card information more private than their home address. But surely there are people like J.D. Salinger who less about the privacy of their credit card number (which is easily changeable) than their home address (which isn't). If someone finds Salinger's address and posts it on the Web, should Salinger be able to demand that Google de-index the page? Why should Google cater to the majority who want to keep their credit card number secret, but not to the minority who care more about keeping their address secret? Another commenter on Matt Cutts's blog post asked:

"hi. I have a question. My mom 'googled' herself and it shows some of her medical problems. She wants/needs these pages removed from search engines."

Again, why shouldn't that be considered at least as private as a credit card number?

And finally, even Google's decision to display an "offensive results" disclaimer, for some results but not for others, raises the same "Where do you draw the line?" questions as the issue of page removal. The Michelle Obama monkey picture gets a disclaimer. But search for 'george w bush' and the first row includes a photoshopped (I think!) image of Bush flipping off the press. Does that warrant a disclaimer as well? (Maybe that's considered less unfair because, even though the picture is fake, it does depict something that actually happened.) The first image result for "bristol palin" is a photo of her engaged in underage drinking — a real photo, but probably unfair to call it the single most relevant photo of her on the Web.

So while Google might consider credit cards and social security numbers and search engine spam to be on one side of a "bright line," and everything else is served up without alteration, I think the line is blurrier than that, for at least those three reasons: (a) credit cards and SSNs are less private than some other that things that Google serves up anyway; (b) Google has unambiguously removed some content that fell outside that bright line, as in the Chester's guide incident, and (c) they make other "slippery slope" judgment calls about search results all the time (as in the question of when to show the disclaimer). So I hope that Google someday comes out with a more complete answer to the question. What is their real policy on what they will remove? The Chester's guide incident — would they do that sort of thing if the same situation came up today, or have their rules changed? If they want to go really deep, then is there a general set of principles from which their rules follow — explaining why, for example, they treat credit card numbers as more private than sensitive medical information? (Google did not respond to my request for comment, either through official channels or the unofficial back channels of friends who work there.)

I hope Google gives an answer some day. Even just to say, "It's a classified internal policy and that's all we're going to tell you." But once and for all, the answer is not "Google doesn't remove content just because it's 'offensive' or 'harmful.'"

Meanwhile, a modest suggestion about the disclaimer displayed above the search results: Put it where people will actually see it, in a separate line below the ads, but above the search results. Right now the link to the disclaimer is displayed as one of three ads across the top, and people don't look at the ads. But hey, people do buy ads, so if you push the disclaimer down a bit where people will read it, you also free up space for 50% more ad revenue!

Microsoft thinks LookSmart looks less smart. securitas writes "Internet search company LookSmart was dropped by Microsoft's MSN service today. MSN has decided not to extend its licensing contract with LookSmart beyond January 2004. The news is devastating for LookSmart since, 'Microsoft accounted for approximately 65 percent of listings revenues, and all of LookSmart's licensing revenue in the second quarter of 2003,' according to a company press release not listed on LookSmart's site. The move comes after LookSmart recently launched its own Overture-style pay-per-click service and indicates Microsoft is close to launching its new search engine technology designed to unseat Google for the search crown. All of this is against the backdrop of acquisitions by rival Yahoo. More coverage at SearchEngineWatch and a Reuters mirror at CNN Money."

They could have fixed this if they'd googled for an answer ... powerg3 writes "This follow-up, explains the Google wackiness posted yesterday. Here's the quote from the Google Weblog: '...when a spam result comes up in a search, Google not only blocks the spam, but every result after it. This means that for searches where spam results manage to rise to the top, very few -- sometimes zero -- results will be returned....It's pretty amazing that such a serious bug made it past Google's tests. It will be interesting to see how quickly it's fixed.'"

Pardon me, mum, can I borrow your hard drive? AndreL writes "The Guardian has an update about the BBC's digital archive plans. They're considering using P2P technology to avoid bandwidth bottlenecks. The bad news: because of technical, financial, and legal problems nothing will happen until 2006 at the earliest."

Please arrive in costume if at all. KTecumseh writes "The list of theaters showing the extended editions of LOTR as been revealed. You can check out the list at the official website, and before you look, pray that you live somewhere close by to take advantage of this once-in-a-lifetime experience. For those that can not make the full 10-hour saga, they are also showing the first two extended additions on different dates, but who wants to miss out on an entire theater of sweaty LOTR fans."

Shamus Arrigan asks plaintively "There is no mention of these ticket sales in any other country. Does anyone know when and where these tickets will be sold at? (Canada especially)"

Wait, are you fellas press? Dangnabbit! ccnull writes "Good news from the inquest against hacker Adrian Lamo. According to a paragraph in a Washington Post column (buried about 3/4 of the way down the story), the FBI appears to be backing off from pursuing reporters' notes in the case. Relevant quote: 'A Justice Department official says the FBI agent "acted out of turn" by not seeking approval from the U.S. attorney's office in Manhattan and Attorney General John Ashcroft's press office. "The agent did not follow standard procedures," the official says. "We're just not going to pursue it. It is the policy of the Justice Department to exhaust all other means before seeking information from members of the media."'"

Eh, what's a few orders of magnitude? Grant hayes writes: "It seems the decimal point in the Mono story you ran is being a bit ambitious. We should be reading Mono 0.28, not Mono 2.8. Check the link below as well as links there to other Mono resources; I see 0.28 throughout." Here's the Mono site -- guilty as charged. Thanks for the correction.

Google's advantage, of course, is that its page-ranking means that the Web sites you find are less likely to be affected by how many keywords they can cram in, and more by which other important sites are linking to them. The theory is that lame sites won't be linked-to by important sites, and that therefore they won't show up high in your search results. And the theory usually works pretty well, which is why Google is my preferred engine.

The GeekPress article says that a search on "Liv Tyler nude" has as its top results some links all going to the same site which has Liv Tyler (allegedly) nude. Well, OK then. If you think that's a problem.

Google's CTO Craig Silverstein comments that that particular search query doesn't, "as far as we can tell, have any good results -- in our spot check, for instance, we couldn't actually find any Web sites that show Liv Tyler in the nude. When there are no good results out there, Google's results can be somewhat arbitrary, so it's not particularly surprising this site was first."

When I randomly checked the names of more popular actresses, plus the word "nude," the supposed-scam in question didn't pull down any especially good hits.

This was confirmed by the adult site itself. When I e-mailed its representative, he claimed their ranking for more popular celebrities like Cindy Crawford and Pamela Anderson were way down the Google list: 22nd, 38th, and worse.

And, protesting the "scam" label, he pointed me to a good article on bridge pages. The technique they're using is a popular method of getting hits which -- as long as the destination pages bear relevance to the search terms, which they here do -- is in the gray area usually considered aggressive self-promotion. It's a trick more or less ignored by the search engines until it's combined with other less-savory tricks.

(That article, and most of searchenginewatch.com, makes for fascinating reading if you're interested in the arms race for your eyeballs being fought between engines and webmasters.)

Also, the adult site operator says his site has gotten only 400 hits a day from all the bridge pages they've set up. It's hard to argue that just a few hundred clicks over to Jane Doe nude represent an extraordinary hijacking of the search term "Jane Doe nude."

Google does refine their algorithm, which incidentally like all search engines' is kept secret to avoid giving Web-spammers an edge. You may remember last year's joke of the search "more evil than Satan" pointing (mistakenly, of course) to Microsoft's homepage. As their founders comment in the recent MIT Technology Review interview, this was a little embarrassing for them, and the engine was tweaked to fix it.

And Google's CTO isn't ruling out more tweaking in the future:

In any case, we know our scoring scheme isn't perfect -- even when the sites in question aren't trying to fool us -- and we're always working to improve it. Often the problem isn't, "Why did this bad site score high?" but rather, "Why did these other good sites score low?"

We're always looking at queries that give strange-looking results to get a better understanding of how our scoring can be improved. Whether the "xnude" queries will result in tweaks to our scoring, I can't say, but we'll certainly be adding them to the test cases we look at.

Short version: the arms race continues; Google still kicks butt.

Google's advantage, of course, is that its page-ranking means that the Web sites you find are less likely to be affected by how many keywords they can cram in, and more by which other important sites are linking to them. The theory is that lame sites won't be linked-to by important sites, and that therefore they won't show up high in your search results. And the theory usually works pretty well, which is why Google is my preferred engine.

The GeekPress article says that a search on "Liv Tyler nude" has as its top results some links all going to the same site which has Liv Tyler (allegedly) nude. Well, OK then. If you think that's a problem.

Google's CTO Craig Silverstein comments that that particular search query doesn't, "as far as we can tell, have any good results -- in our spot check, for instance, we couldn't actually find any Web sites that show Liv Tyler in the nude. When there are no good results out there, Google's results can be somewhat arbitrary, so it's not particularly surprising this site was first."

When I randomly checked the names of more popular actresses, plus the word "nude," the supposed-scam in question didn't pull down any especially good hits.

This was confirmed by the adult site itself. When I e-mailed its representative, he claimed their ranking for more popular celebrities like Cindy Crawford and Pamela Anderson were way down the Google list: 22nd, 38th, and worse.

And, protesting the "scam" label, he pointed me to a good article on bridge pages. The technique they're using is a popular method of getting hits which -- as long as the destination pages bear relevance to the search terms, which they here do -- is in the gray area usually considered aggressive self-promotion. It's a trick more or less ignored by the search engines until it's combined with other less-savory tricks.

(That article, and most of searchenginewatch.com, makes for fascinating reading if you're interested in the arms race for your eyeballs being fought between engines and webmasters.)

Also, the adult site operator says his site has gotten only 400 hits a day from all the bridge pages they've set up. It's hard to argue that just a few hundred clicks over to Jane Doe nude represent an extraordinary hijacking of the search term "Jane Doe nude."

Google does refine their algorithm, which incidentally like all search engines' is kept secret to avoid giving Web-spammers an edge. You may remember last year's joke of the search "more evil than Satan" pointing (mistakenly, of course) to Microsoft's homepage. As their founders comment in the recent MIT Technology Review interview, this was a little embarrassing for them, and the engine was tweaked to fix it.

And Google's CTO isn't ruling out more tweaking in the future:

In any case, we know our scoring scheme isn't perfect -- even when the sites in question aren't trying to fool us -- and we're always working to improve it. Often the problem isn't, "Why did this bad site score high?" but rather, "Why did these other good sites score low?"

We're always looking at queries that give strange-looking results to get a better understanding of how our scoring can be improved. Whether the "xnude" queries will result in tweaks to our scoring, I can't say, but we'll certainly be adding them to the test cases we look at.

Short version: the arms race continues; Google still kicks butt.