Slashdot Mirror
Is Microsoft Crawling Google?
triplecoil writes "Jason Dowdell over at WebProNews has written a piece questioning a tactic Microsoft might be using to beef up its new search engine. He thinks they might be dipping into Google's results to supplement its own. Dowdell likens it to leaving your garbage on the curb--anyone could conceivably go through it and take whatever is there for their own."
If you've been watching the logs to your site lately Microsoft has been RAPING most servers. Most crawlers will pick through pages with large lists 1 at a time, then come back every hour or so.
MSN starting last week has been pulling EVERY LINK in sequence from my site. Even the larger Artist Index pages of my site.
Seriously, I've had this same spider on my site for about 36 hours now.
Habbit = What a priest wears
Habit = A regular behavior for a person/thing
Now try it on google and you get articles about the 'more evil that....' debate.
:)
I know which search engine I'm sticking with
It would be easy for Google to insert a small fraction of non-sequiturs in the results, look at Microsoft's search results, and then sue for misuse. Even if MSFT uses random proxies to avoid detection, it cannot manually recheck all the hits to make sure they are correct (if they could, they had the resources to check all the sites, then they not need to crawl Google. A few made-up sites or inappropriate search hits would be enough to establish a pattern of abuse.
Two wrongs don't make a right, but three lefts do.
pritty funny :)
e ngine.ms.idg/ / 2167621
but it seems like google started it several years ago.
http://www.cnn.com/TECH/computing/9911/15/search.
and
http://searchenginewatch.com/sereport/article.php
btw, it doesen't seem to work on google anymore...
All indicators show that the human race is selectively breeding itself for stupidity.
Google Terms of Service
... You may not take the results from a Google search and reformat and display them, or mirror the Google home page or results pages on your Web site. You may not "meta-search" Google. If you want to make commercial use of the Google Services, you must enter into an agreement with Google to do so in advance ..."
"
Yes it does.
From Googles Privacy Center (http://www.google.com/terms_of_service.html):
Personal Use Only
The Google Services are made available for your personal, non-commercial use only. You may not use the Google Services to sell a product or service, or to increase traffic to your Web site for commercial reasons, such as advertising sales. You may not take the results from a Google search and reformat and display them, or mirror the Google home page or results pages on your Web site. You may not "meta-search" Google. If you want to make commercial use of the Google Services, you must enter into an agreement with Google to do so in advance. Please contact us for more information.
Find a link, fine
Follow the link, fine
Spider the link, not fine - google's Robots.txt does not give them permission to.
Exigo spamos et dona ferentes
IANAL but I would see this as falling under fair use.
1) the LoC is not profitting from your works nor is it re-using them (with the exception of providing an archive to others, see next item).
2) the LoC regularly tells people requesting copies of their information to first obtain permission from the copyright holder (in other words, as with any library, you can browse but you can't copy without permission and copy permission does not equal permission to reuse in a commercial work).
3) Copy protection schemes require active protection to fall under the DMCA, even if it is so simple that anyone can defeat it. Robots.txt is -passive- protection because you have to purposefully search for the file and then purposefully utilize it. To be active protection the document should not come up without the viewer (or blocked viewer) performing some form of action. When someone/something visits an unprotected public web page there is not a way for your web server to invoke the robots.txt file, therefore it is not an active mechanism.
It is more productive to voice thoughtful opinions (reply) than to judge (moderate) others.
Remove yourself from google
"Note: If you believe your request is urgent and cannot wait until the next time Google crawls your site, use our automatic URL removal system. In order for this automated process to work, your webmaster must first insert the appropriate meta tags into the page's HTML code. "
Have you ever been to a turkish prison?
And MSN crawling Google's site is really no different. As long as the Google data is on a public server, it is fair game to crawl.
His thought process probably started here
https://www.eff.org/https-everywhere
so if the msn bot does what they say it doesn't do what it's supposed to do.
To remove all the images on your site from our index, place the following robots.txt file in your server root: /
User-agent: Googlebot-Image
Disallow:
That should work? No?
I touch computers in naughty places